This update was made possible by the discussions, implementation improvements, and bug fixes from numerous contributors over the past several years. We would like to take this opportunity to express our gratitude.

Requirements for the Optuna Default Sampler

Optuna is not only the de facto standard library for hyperparameter optimization but is also utilized across diverse domains beyond machine learning, such as materials science and structural design. To support such a wide range of use cases, the default sampler is required to offer not only optimization performance but also usability in various respects. In this update, we considered the following points:

Define-by-Run API

Optuna employs a Define-by-Run style API, where the search space is defined during execution. While this API allows users to write code in a simple and intuitive manner, it requires careful implementation in the sampler. For algorithms that consider relationships between parameters, the search space must be inferred from past history, and it is desirable to maintain a certain level of optimization performance even if that inference is incorrect.

Discrete and Conditional Search Spaces

Optuna also supports integer and categorical search spaces. It also handles complex search spaces where hyperparameters may or may not exist depending on certain conditions. The default sampler must be able to handle not only continuous search spaces but also such spaces.

Asynchronous Optimization

Optuna enables multiple workers to process Trials asynchronously. In asynchronous optimization, the sampler must handle situations where some Trials have not yet returned results, or where the results arrive out of order.

Pruning and Constraint-Based Optimization

Optuna must also be able to handle more advanced optimization problems, such as cases where Trials are pruned by the Pruner mid-evaluation or cannot be evaluated because they do not satisfy constraints.

Dependency Libraries

Since Optuna is used in various environments, it is necessary to minimize the number of dependency libraries to improve convenience. For example, if a heavy library like PyTorch is included as a dependency, some users may encounter version conflicts or memory limitations.

Speed

Hyperparameter optimization for machine learning rarely involves a large number of Trials, but since Optuna is used across a wide range of fields, use cases with several thousand Trials or more are not uncommon. The default sampler must run sufficiently fast even in such settings.

To meet these various needs, a default sampler that simply has high optimization performance is not enough. TPE, adopted as the default in Optuna, is an algorithm that is well suited to implementations that satisfy these conditions. Roughly speaking, TPE is an algorithm that partitions past Trials into “good” and “bad” and uses the resulting probability distributions to propose promising points next. In other words, as long as the conditions of “being able to partition” and “being able to derive a probability distribution” are met, it can flexibly handle various problems. Considering this advantage, we examined whether improvements could be made specifically for TPE during the v5 update.

Below, we introduce the efforts we undertook to update the default sampler.

Efforts Toward the Update

This is not the first time we have worked on updating the default sampler; we also attempted improvements during the v3 major release. Unfortunately, this did not result in an update to the default sampler at the time, but we learned the following lessons:

1. In benchmarks, there must be clear improvements significant enough for developers to reach a consensus.
2. Implementation bugs that affect performance must be eliminated in advance.

The second point is particularly important. Since Optuna solves various optimization problems across diverse environments, users may actually encounter bugs even if they occur only in rare cases. Furthermore, if benchmarks are conducted while these issues remain, it becomes difficult to interpret the resulting performance changes.

From v3 through v5, we have resolved various issues. Additionally, since the sampler execution speed had been a problem in multi-objective optimization, steady improvements have been accumulated. Below, we introduce some of these.

Bug Fixes and Optimization Performance Improvements

• #4073 (v3.1): Trials currently running end up on the “below” side in constant_liar
• #4462 (v3.2): Bug when combining multivariate and constant_liar
• #6189 (v4.5): Optimization performance may degrade when combining multivariate and constant_liar
• #6258 (v4.7): Issue related to log-density calculation
• #6574 (v4.9): Prior distribution is mixed into bandwidth calculation

Speed Improvements

• #5303 (v4.0): Vectorization of 2D hypervolume calculation
• #5346 (v4.0): Improvement of HSSP computational complexity
• #5355 (v4.0): Lazy evaluation of HSSP
• #5424, #5591 (v4.0): WFG speed optimization
• #6112 (v4.4): Dedicated 3D implementation
• #6194 (v4.5): Speed optimization for ndtri_exp
• #6200 (v4.5): Speed optimization for the error function

Changes in v5

The following changes were made in v5. All of these were either shown to improve optimization performance in published papers or were previously expected by the development team to yield performance gains.

Changed the default sampler for multi-objective optimization to TPESampler

While multi-objective TPE was reported in the paper [1] to outperform NSGA-II, its execution speed was initially a bottleneck that prevented it from being adopted as the default. This issue has been resolved through years of speed optimization; therefore, in v5, TPESampler is now the default for both single- and multi-objective optimization.

Multivariate TPE enabled by default

Multivariate TPE captures interactions between variables by modeling their joint distribution, rather than treating each variable independently. [2] reports that it performs better when combined with Optuna’s bandwidth selection, and since our replication studies confirmed a similar trend, we have enabled this feature by default. Note that for multi-objective optimization, no performance improvement was observed in benchmarks, so independent TPE will be used.

Constant Liar enabled by default

In TPE, performing parallel optimization without specific adjustments can lead to a problem where each worker references the same completed Trials, resulting in the proposal of nearly identical parameters. With Constant Liar enabled, the proposal of candidate points close to currently running Trials is suppressed, which is expected to improve performance in parallel optimization.

Optimization Performance

The changes in v5 were guided by papers and our past experiments, and we proceeded while verifying that their findings could be reproduced. The figures shown below illustrate the win rates for each benchmark problem set under the default settings of v4.9 and v5.0, with the number of Trials plotted on the x-axis.

For single-objective optimization, we referred to the recommended settings in [2] and confirmed that the performance trends were reproduced. The figures show the performance improvement achieved by Multivariate TPE (1-a), in addition to the performance improvement in batch optimization via Constant Liar (1-b). On the other hand, as also reported in [2], a performance degradation due to Multivariate TPE was observed in HPOBench (1-c). Whether performance improves depends on the problem, which is to some extent unavoidable considering the No Free Lunch theorem; therefore, we implemented this change after determining that improvements are expected for many of Optuna’s use cases.

For multi-objective optimization, we confirmed that the change from NSGA-II to TPE led to significant improvements, particularly around 100 Trials. Furthermore, since experiments revealed that Multivariate TPE tends to degrade performance in multi-objective optimization, we decided against adopting it, unlike for single-objective optimization.

That said, benchmarks alone cannot account for workloads across various fields. However, since many users were already using these three options, we determined that while not all workloads would necessarily improve, there was likely no significant risk, and thus decided to adopt them.

Looking Ahead

In this major release, we have updated the default sampler for the first time in a while, but we believe there is still room for improvement. While we focused the discussion on TPE this time, we would like to explore other samplers such as GPSampler and CmaEsSampler, as well as the wide variety of algorithms registered on OptunaHub.

At the same time, as evidenced by the fact that this was the first update in several years, determining what the default for Optuna should be has been a very difficult problem, and that will remain the case. It is no longer possible to fully grasp exactly what kinds of problems users are solving and with which algorithms. If you have examples of “this worked well/didn’t work” or “this algorithm was effective,” we would greatly appreciate it if you could publish and share them, as it will aid future development. We look forward to your continued contributions to Optuna in various ways.

References

[1] Ozaki, Y., Tanigaki, Y., Watanabe, S., Nomura, M., & Onishi, M. (2022). Multiobjective Tree-Structured Parzen Estimator. Journal of Artificial Intelligence Research, 73, 1209–1250.

[2] Watanabe, S. (2023). Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance. arXiv preprint arXiv:2304.11127.

Improving the Default Optimization Algorithm in Optuna v5 was originally published in Optuna on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article introduces PedAnovaImportanceEvaluator the new default hyperparameter importance evaluator in Optuna v5 and Rustuna.

Hyperparameter importance (HPI) quantifies how strongly each hyperparameter influences the observed objective values and plays an important role in hyperparameter optimization. HPI helps interpret optimization results, identify important design choices, detect irrelevant or redundant hyperparameters, and guide subsequent search or manual tuning.

While Optuna previously used f-ANOVA [2] as its default HPI evaluator, PED-ANOVA [4] becomes the new default in Optuna v5, offering several practical advantages:

• it is substantially faster than f-ANOVA;
• requires no additional dependencies beyond Optuna’s minimal installation; and
• can support conditional search spaces as well as multi-objective optimization.

In particular, support for conditional search spaces is enabled by a new method we developed, which was accepted at KDD 2026, one of the top conferences in the machine learning and data mining field. For more details, please see our paper [1]:

• Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces

How to Use PED-ANOVA in Optuna & Rustuna

Basic Usage

PED-ANOVA can be used with Optuna or Rustuna without any additional installation:

# Using Optuna:
uv add "optuna>=5.0.0"

# Using Rustuna:
uv add "rustuna>=0.1.0"

# If you use pip:
pip install "optuna>=5.0.0"
# or
pip install "rustuna>=0.1.0"

Once the installation is done, you can easily use PED-ANOVA by passing a study containing completed trials to optuna.importance.get_param_importances:

import optuna
# If you use Rustuna:
# import rustuna as optuna


def objective(trial: optuna.Trial) -> float:
    x = trial.suggest_float("x", -5.0, 5.0)
    y = trial.suggest_float("y", -1.0, 1.0)
    return x**2 + y**2

sampler = optuna.samplers.RandomSampler(seed=42)
study = optuna.create_study(sampler=sampler, direction="minimize")
study.optimize(objective, n_trials=100)

importance = optuna.importance.get_param_importances(study) # Use PedAnovaImportanceEvaluator as default
print(importance) # {'x': 0.9773158726469768, 'y': 0.022684127353023158}

If you are using Optuna, you can also visualize the importance scores with the following code. Note that Plotly or Matplotlib must be installed separately for visualization.

# Use Plotly
fig = optuna.visualization.plot_param_importances(study)
fig.show()

# Use Matplotlib
import matplotlib.pyplot as plt
ax = optuna.visualization.matplotlib.plot_param_importances(study)
plt.show()

Interpretation of Importance Scores

HPI is defined based on how much each hyperparameter contributes to the variance of the objective values. PedAnovaImportanceEvaluator estimates this importance within a specified performance region using two key arguments, target_quantile and region_quantile. The returned importance indicates how important each hyperparameter is for achieving performance within the top target_quantile fraction, when focusing on the region corresponding to the top region_quantile fraction of the observed performance.

evaluator = optuna.importance.PedAnovaImportanceEvaluator(target_quantile=0.1, region_quantile=1.0) # The default settings
importance = optuna.importance.get_param_importances(study, evaluator=evaluator)
print(importance)

Experimental Results

In this section, we present experimental results on a real-world benchmark problem to illustrate how PED-ANOVA behaves in practice. For a more comprehensive empirical analysis of PED-ANOVA’s handling of conditional search spaces, including performance comparisons with baseline methods, please refer to our paper [1].

Experimental Setup

Benchmarking Problem. We use instance 1053 of the rbv2_super scenario from YAHPO Gym [3]. This benchmark represents a model selection problem over six machine learning algorithms: approximate k-NN (aknn), a regularized linear model (glmnet), random forest (ranger), decision tree (rpart), support vector machine (svm), and XGBoost (xgboost). The algorithm is selected by the categorical hyperparameter learner_id, and only the hyperparameters associated with the selected algorithm are active, naturally forming a conditional search space.

Evaluation Settings. We use a sample size of 𝑁 = 1000 and maximize the accuracy (acc) as the objective. We use target_quantile = 0.1 and region_quantile = 1.0. We use Python version 3.13.11 and Optuna v5.0.0. Please refer to our GitHub repository for the experimental code.

Results and Discussion

Figure 1 shows the HPIs estimated by PED-ANOVA. We observe that trainsize and learner_id, as well as several SVM- and XGBoost-related hyperparameters, have relatively large importance scores. This is consistent with the actual objective values, where SVM and XGBoost achieve higher performance than the other learners, suggesting that PED-ANOVA successfully captures the importance of the learner choice.

We further calculate the correlation between the maximum objective value achieved by each learner and its estimated importance. PED-ANOVA shows the highest correlation among the evaluated methods, outperforming naive extensions of existing HPI approaches to conditional search spaces. Moreover, unlike these baselines, PED-ANOVA does not exhibit unstable behavior such as negative correlations. For further details, please refer to our paper.

Runtime Comparison. Figure 2 compares the runtime of PED-ANOVA in Optuna v5, PED-ANOVA in Rustuna, and f-ANOVA, the previous default evaluator in Optuna, on the same YAHPO Gym benchmark problem. Optuna’s PED-ANOVA is dramatically faster than f-ANOVA, delivering speedups of roughly 180× to 1,600× across the tested trial counts. Rustuna’s PED-ANOVA takes this further, running approximately 2–4× faster than Optuna’s implementation.

Conclusion

In this article, we introduced PED-ANOVA, which is now the default HPI evaluator in Optuna v5 and Rustuna. PED-ANOVA enables fast and practical HPI evaluation while supporting conditional search spaces and multi-objective optimization.

Please give it a try!

References

[1] Baba, K., Ozaki, Y., and Watanabe, S. Conditional PED-ANOVA: Hyperparameter importance in hierarchical & dynamic search spaces. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, pp. 57–68. Association for Computing Machinery, 2026. ISBN 9798400722592. doi: 10.1145/3770855.3817758.

[2] Hutter, F., Hoos, H., and Leyton-Brown, K. An efficient approach for assessing hyperparameter importance. In Proceedings of the 31st International Conference on Machine Learning, pp. 754–762. PMLR, 2014.

[3] Pfisterer, F., Schneider, L., Moosbauer, J., Binder, M., and Bischl, B. YAHPO Gym — an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization. In Proceedings of the First International Conference on Automated Machine Learning, volume 188, pp. 3/1–39. PMLR, 2022.

[4] Watanabe, S., Bansal, A., and Hutter, F. PED-ANOVA: Efficiently quantifying hyperparameter importance in arbitrary subspaces. In Proceedings of the 32nd International Joint Conference on Artificial Intelligence, pp. 4389–4396, 2023.

[Optuna v5 & Rustuna] PED-ANOVA Is Now the Default Hyperparameter Importance Evaluator was originally published in Optuna on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article introduces the q-batch Monte Carlo-based acquisition functions added to GPSampler in Optuna v5.0.

GPSampler is a sampler for Gaussian process-based Bayesian optimization. It delivers particularly strong performance on continuous-function optimization, and since its introduction in Optuna v3.6, it has been extended and improved in various ways, including support for constrained multi-objective optimization. Compared with other samplers, generating a single candidate point takes longer, but its strong optimization performance makes it especially effective for problems with high evaluation cost.

On the other hand, up through Optuna v4.7, GPSampler performed Gaussian process regression using only completed evaluations. As a result, in batch optimization, the next candidate points were selected from nearly identical regression models based on the same evaluated data, which led to nearly identical points being proposed repeatedly.

To address this issue, Optuna v4.8 and v4.9 introduced improved parallelization using Constant Liar and Kriging Believer. These methods prevent the same point from being selected repeatedly by assigning temporary values, such as the best value observed so far, to points that are still under evaluation.

However, these approaches require heuristic imputation of values for running points, leading to issues such as the lack of clear guidance for choosing the imputed values and limitations in performance. To address these challenges, Optuna v5.0 introduced Monte Carlo-based q-batch acquisition functions [1], further improving the parallel optimization performance of GPSampler.

With this approach, predicted objective values for running trials are sampled in a Monte Carlo manner from the posterior distribution of the Gaussian process regression model. The improvement obtained in each sample is then evaluated, and the acquisition function value is computed by averaging these improvements. In Optuna, this is implemented by sequentially optimizing the q-batch acquisition function and selecting candidates one by one. These features do not require any special arguments and are automatically enabled when parallel optimization is run as usual.

Below, we first explain the outline of the newly introduced algorithm using qLogEI, which is used for single-objective unconstrained optimization, and then present benchmark results.

How qLogEI Works

In this section, we focus on the algorithm for single-objective unconstrained optimization (qLogEI). For implementation details, please refer to PR #6640.

In qLogEI, objective values at the currently running points and a new candidate point are sampled multiple times using QMC from the Gaussian process posterior conditioned on completed trials. For each sample, the amount by which the value exceeds the threshold is evaluated as the log improvement, and the maximum log improvement is taken over the set consisting of the running points and the candidate point. Finally, qLogEI is computed by aggregating these values over the sample dimension in log space (see Figure 1).

In this way, qLogEI does not simply evaluate how good a single candidate point is. Instead, it evaluates how promising the entire set of q points is, including trials that are already running. This enables qLogEI to select the next candidate while accounting for other running trials in parallel optimization.

In constrained optimization, the improvement at each point in each sample is multiplied by the Probability of Improvement (PI). In multi-objective optimization, the hypervolume improvement with respect to the Pareto front formed by completed and running points is computed for each sample. For details, please refer to the following PRs:

• #6764: Support for single-objective constrained optimization (qLogCEI)
• #6792: Support for multi-objective unconstrained optimization (qLogEHVI)
• #6804: Support for multi-objective constrained optimization (qLogCEHVI)

Benchmark Results

Next, we conducted benchmarks to evaluate the optimization performance of GPSampler with the newly introduced Monte Carlo-based q-batch acquisition functions. We compared the following three methods:

• GPSampler in v5.0 (parallelization using qLogEI)
• GPSampler in v4.9 (parallelization using Constant Liar)
• TPESampler (Optuna’s default sampler)

Benchmark problems and implementation details: we used BBOB problems with function_id = [2, 3, 4, 5] [2]. We considered a five-worker parallel optimization setting, in which candidate points were generated sequentially while four points were already under evaluation. We ran 10 experiments with different random seeds and computed the mean and standard error. The first 10 trials, corresponding to n_startup_trials, were excluded from the plots.

Computing environment: the benchmarks were run on Ubuntu 24.04 on an x86_64 architecture. The CPU used was an Intel 12th Gen Core i7–1260P (12 cores, 16 threads, up to 4.70 GHz).

The results are shown below (Figure 2).

Compared with v4.9, the optimization performance of v5.0 improved substantially, with a tendency to converge in fewer trials. Compared with TPESampler, GPSampler v5.0 also showed better performance in many problem settings.

For benchmarks on constrained optimization and multi-objective optimization, please refer to the respective PRs.

Conclusion

In Optuna v5.0, we improved the parallel optimization performance of GPSampler by introducing Monte Carlo-based q-batch acquisition functions. This makes it possible to generate candidate points while accounting for running trials, without assigning heuristic values to points that are still under evaluation. GPSampler with this feature is especially useful for batch optimization problems with high evaluation cost. Give it a try!

Reference

[1] Ament, S., Daulton, S., Eriksson, D., Balandat, M., & Bakshy, E. (2023). Unexpected improvements to expected improvement for Bayesian optimization. Advances in Neural Information Processing Systems, 36, 20577–20612.

[2] The blackbox optimization benchmarking (bbob) test suite (https://hub.optuna.org/benchmarks/bbob/)

Improving Parallel Optimization Performance in GPSampler with q-Batch Acquisition Functions was originally published in Optuna on Medium, where people are continuing the conversation by highlighting and responding to this story.

Inspired by the success of tools like Ruff, Polars, and Pydantic V2, which achieved dramatic performance improvements by leveraging Rust, we have developed and released Rustuna, a faster Optuna implementation built with Rust. In this post, we introduce Rustuna’s key features, how to use it, and our vision for its future.

• GitHub: https://github.com/optuna/rustuna/
• Documentation: https://rustuna.readthedocs.io/

Alongside Rustuna, we are also excited to announce Optuna 5.0, the fifth major release of Optuna. For full details on the new features and improvements, check out the Optuna 5.0 Release Blog.

Getting Started with Rustuna

Rustuna provides high compatibility with Optuna. If you are already familiar with Optuna APIs, there is no need to learn a new API, so you can transition smoothly. Here is how to get started:

$ pip install rustuna

import rustuna

# 1. Define an objective function which returns a value you want to minimize.
def objective(trial: rustuna.Trial) -> float:
    # 2. Define the search space via Suggest APIs.
    x = trial.suggest_float("x", -10, 10)
    y = trial.suggest_float("y", -10, 10)
    return (x - 2) ** 2 + (y + 5) ** 2

# 3. Create a study which manages each experiment.
study = rustuna.create_study()

# 4. Evaluate your objective function.
study.optimize(objective, n_trials=100)

# 5. Print the best evaluation parameters.
print(study.best_trial)

While Rustuna is not yet as feature-rich as Optuna, it already includes the core functionality most users need. Furthermore, except for a few specific features, Rustuna does not require Python dependencies. This minimal footprint helps mitigate risks associated with the recent surge in supply chain attacks. Even if execution speed or memory usage has never been a bottleneck for you, we encourage you to give it a try.

If you need features that Rustuna does not support, you can still leverage Optuna’s functionality. While certain use cases may not fully benefit from Rustuna’s speed advantages, this allows you to seamlessly use Optuna features, such as its visualization modules, after executing an optimization. For details, please refer to the official Rustuna documentation.

How Rustuna Got So Fast

Rustuna was built with a strong focus on its execution speed. Figure 1 compares the execution speeds of the sampling algorithms provided by Rustuna and Optuna (measured on a MacBook with M4 Pro).

TPE and MOTPE (Multi-Objective TPE) are the default optimization algorithms used in Optuna. With TPE, the computation time required to sample parameters increases proportionally with the number of trials. While Optuna recommended using TPE primarily for runs under 1,000 trials, Rustuna’s high-speed TPE implementation can complete 10,000 trials, ten times as many, in under a minute. This performance gap becomes even more pronounced when leveraging multi-threading (Table 1).

Algorithms like CMA-ES and NSGA-II also achieve massive speedups, but not simply because they were rewritten in Rust. Algorithmic theory dictates that sampling time for these methods should remain constant relative to trial count; however, due to architectural trade-offs, Optuna’s execution speed degrades as trial counts increase. Rustuna achieves its speed gains by combining two approaches: an optimized Rust implementation and a refined algorithmic architecture.

Memory-efficient Storage Implementation

Rustuna is also focused on optimizing memory consumption. In addition to performance gains achieved through lower-level architectural and implementation tuning, Rustuna differs significantly from Optuna by providing a feature to discard optimization histories that are no longer needed during optimization.

Unlike TPE or Gaussian Process-based Bayesian optimization algorithms, methods such as CMA-ES and NSGA-II only reference a subset of evaluation histories while running. However, Optuna retains all evaluation histories in memory indefinitely, causing both memory usage and execution time to degrade as the optimization progresses. By using the apply_discard option provided across Rustuna’s storage classes, unnecessary trial history is automatically purged from memory during the run.

import rustuna

def objective(trial: rustuna.Trial) -> float:
    x = trial.suggest_float("x", -10, 10)
    trial.set_user_attr("large_text", "a" * 10000)
    return (x - 2) ** 2

# Set apply_discard=True to enable feature to discard trials.
storage = rustuna.storages.JournalFileStorage("./rustuna-journal.db", apply_discard=True)

sampler = rustuna.samplers.CmaEsSampler()
study = rustuna.create_study(sampler=sampler, storage=storage, study_name="example")
study.optimize(objective, n_trials=100000)

While discarded during optimization, there are cases where you still want to analyze the complete evaluation history afterward. By loading the storage file without specifying the apply_discard option, the full history, including all previously discarded trials, can be loaded:

storage = rustuna.storages.JournalFileStorage("./rustuna-journal.db")
study = rustuna.load_study(storage=storage, study_name="example")
print(study.best_trial)

We also validated the impact of this feature in a real-world production service. Matlantis Corporation utilizes a custom NSGA-II-based method built on Optuna for their crystal structure prediction platform, Matlantis CSP (see: Crystal Structure Prediction Using Optuna in Matlantis CSP). Figure 2 shows the impact on memory usage and execution speed after migrating to Rustuna.

With the Optuna implementation, memory consumption increases as evaluation histories accumulate over time. Furthermore, because their custom NSGA-II algorithm sequentially scans the complete evaluation history to extract necessary values, processing times degrade as the list grows.

In contrast, Rustuna’s apply_discard option discards unneeded trials, keeping memory growth minimal and preventing execution slowdowns even over long runs. This enables larger trial counts and greater parallelism. Based on these validation results, active development is now underway to fully integrate Rustuna into production.

What’s Ahead

Below is a look at our upcoming development roadmap for Rustuna. While Rustuna does not yet offer every capability available in Optuna, it already supports a wide range of essential features, including key optimization algorithms, storage backends, and hyperparameter importance calculations, making it fully capable of handling the vast majority of use cases.

To further improve usability and expand possibilities, we plan to focus on the following enhancements:

1. Improving Stability: Rustuna is still an emerging project, and new bugs are expected following this initial release. We plan to continuously refine and improve stability based on community feedback.
2. Seamless Optuna Integration: Rustuna is not intended as a complete replacement for Optuna, nor is its goal to match Optuna’s vast feature set. Instead, it is designed to work in tandem with Optuna. We will continue strengthening integration so users can invoke Optuna features even more effortlessly.
3. Multi-Language Support: Because Rustuna is implemented in Rust, it can easily target other languages via C APIs or WebAssembly. While development has focused primarily on Python bindings so far, we plan to explore support for Node.js and other ecosystems.

Additionally, we plan to bring the architectural insights gained from Rustuna back into Optuna itself. Because Optuna supports a large ecosystem, maintaining backward compatibility is critical, requiring a careful, step-by-step approach to structural updates. However, several design changes introduced in Rustuna, such as bulk-insert storage APIs, trial queues, and the apply_discard option, offer clear benefits for Optuna’s performance. We look forward to gradually porting these improvements back to Optuna while ensuring seamless compatibility.

Conclusion

In this post, we covered the background behind Rustuna’s development, how to get started, and its key advantages over Optuna.

For Rustuna to evolve into a mature and reliable software project, real-world usage and community feedback are vital. We invite you to give Rustuna a try, and if you encounter any issues or have suggestions, please feel free to open an issue on GitHub!

Rustuna: A Faster Optuna Implementation in Rust was originally published in Optuna on Medium, where people are continuing the conversation by highlighting and responding to this story.

Since its public release in 2018, Optuna has grown through extensive feature development into software used by a large community. With over 14,000 GitHub stars, more than 16 million monthly downloads, over 32,000 dependent projects, and over 16,000 paper citations, it has earned the support of many users over eight years and become one of the most widely used black-box optimization OSS libraries.

Although Optuna was originally released as a hyperparameter optimization framework for machine learning, it has increasingly been adopted in recent years as a more general-purpose black-box optimization framework, expanding into diverse domains beyond machine learning such as materials science and structural design.

To help Optuna solve even more real-world problems, our development team has worked hard on a high-performance next version. Today, we are releasing Optuna v5, a new major version of Optuna, together with Rustuna, a black-box optimization framework offering a fast Optuna-compatible API implemented in Rust.

• Optuna v5: This is the direct successor to the current Optuna line. It includes several changes befitting a major update: the first default sampler change since Optuna v1.5, a new API for constrained optimization, a change to the default parameter importance algorithm, and enhancements to the Gaussian-process-based optimization algorithm.
• Rustuna: Optuna’s design assumes hyperparameter optimization of machine learning models, where model training dominates runtime. In diverse use cases beyond machine learning, however, the situation differs, and sampling or storage can become the bottleneck. Rustuna is a fast, memory-efficient, Optuna-compatible framework implemented in Rust to address these performance challenges.

This post introduces the major updates in Optuna v5 individually. Rustuna is covered in detail in a separate release blog post.

Key Changes in Optuna v5

Major Update to Optuna’s Default Optimization Algorithms

Optuna’s default samplers have long been the Tree-Structured Parzen Estimator (TPE) for single-objective optimization and Non-dominated Sorting Genetic Algorithm II (NSGA-II) for multi-objective optimization, with their internal settings left largely unchanged for years. In Optuna v5, we improved the default optimization algorithms based on research and benchmark results accumulated over the past several years (Figure 1).

1. Switched the default multi-objective sampler to TPESampler: Multi-objective TPE was reported in [1] to outperform NSGA-II, but at the time it could not be adopted as the default because its execution speed was a bottleneck. Years of speed improvements have since resolved this issue, so in v5, TPESampler becomes the default for both single-objective and multi-objective optimization.
2. Enabled Multivariate TPE by default: Instead of treating each parameter independently as in the conventional approach, Multivariate TPE captures interactions between parameters by modeling their joint distribution. Multivariate TPE was reported in [2] to have an advantage when combined with Optuna’s bandwidth selection, and our own follow-up experiments confirmed a similar trend in single-objective optimization, so we enabled this feature by default. For multi-objective optimization, since we observed no performance improvement in benchmarks, we will continue using independent TPE as before.
3. Enabled Constant Liar by default: Without special handling, parallel optimization with TPE causes each worker to reference the same set of completed trials, resulting in nearly identical parameter proposals. With Constant Liar enabled, proposals near trials that are still being evaluated are suppressed, which is expected to improve performance in parallel optimization.

Details of the follow-up experiments and benchmarks conducted for the default sampler changes will be shared in a separate blog post.

New API for Constrained Optimization

Optuna has supported constrained optimization since v2.4, but this was positioned as a feature of certain samplers. In Optuna v5, constrained optimization has been elevated from a sampler-level feature to a core feature of Optuna, available through a more natural API. Going forward, constraints can be set simply by calling trial.set_constraint() inside the objective function, and constraint values can be retrieved via trial.constraints.

import optuna

def objective(trial):
    trial.set_constraint("c0", c0)
    trial.set_constraint("c1", c1)
    …

sampler = optuna.samplers.TPESampler()
study = optuna.create_study(sampler=sampler)
study.optimize(objective)
print(study.best_trial.constraints)

PED-ANOVA as the Default for Parameter Importance Calculation

Parameter importance quantifies how much each parameter affects the observed objective function value. It is highly useful for interpreting optimization results, identifying key design decisions, detecting irrelevant or redundant parameters, and guiding subsequent search processes or manual tuning.

Optuna used f-ANOVA [3] as its default parameter importance algorithm, but starting with Optuna v5, it uses PED-ANOVA [4] as the default instead. This brings several benefits: dramatically faster parameter importance computation compared to f-ANOVA, usability with only Optuna’s standard dependencies, and newly developed support for conditional search spaces and multi-objective optimization. In particular, the conditional search space support we developed was accepted at KDD 2026, one of the top conferences in the machine learning field. See the paper below [5] for details:

• Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces

Parameter importance using PED-ANOVA in Optuna can be used as follows:

import optuna

def objective(trial: optuna.Trial) -> float:
    model = trial.suggest_categorical("model", ["nn", "tree"])
    if model == "nn":
        n_layers = trial.suggest_int("n_layers", 1, 5)
        return train_nn(n_layers)
    else:
        min_split_gain = trial.suggest_float("min_split_gain", 0.0, 1.0)
        return train_tree(min_split_gain)

study = optuna.create_study(direction='minimize')
study.optimize(objective, n_trials=100)

importance = optuna.importance.get_param_importances(study)
print(importance)

fig = optuna.visualization.plot_param_importances(study)
fig.show()

Further details will be covered in a separate blog post. Please try the new Optuna v5 parameter importance calculation.

GPSampler

GPSampler is a sampler that performs Bayesian optimization based on Gaussian processes. It delivers strong optimization performance, especially on problems involving continuous parameters, and is well suited to optimization problems where each evaluation is time-consuming or costly.

Through Optuna v4.7, GPSampler performed Gaussian process regression using only completed trials. As a result, trials still under evaluation were not reflected in the regression model, causing nearby candidate points to be proposed repeatedly in batch optimization.

Optuna v4.8 introduced Constant Liar to address this issue. With Constant Liar, provisional objective function values are assigned to points still under evaluation and treated as observed points, suppressing subsequent candidate points from overlapping with those under evaluation. However, this approach had the drawback that the values assigned to points under evaluation were determined heuristically.

To address this, Optuna v5 introduces a Monte Carlo-based acquisition function that accounts for points under evaluation. This method draws multiple samples of the objective function value at the points under evaluation from the Gaussian process posterior distribution. It then computes the improvement for each sample set and derives the final acquisition function value as their average. This allows candidate points to be proposed while accounting for predictive uncertainty, without assigning a single heuristic value to points under evaluation. Below is the optimization performance assuming 5-way parallel optimization, proposing candidate points in sequence (Figure 2). It clearly shows higher optimization performance than the conventional Constant Liar approach.

Simultaneously Releasing “Rustuna,” a Fast Rust Implementation of Optuna

Optuna is now used for a wide variety of purposes beyond hyperparameter optimization. This has surfaced new challenges. For example, in use cases such as crystal structure search, it is not uncommon for the number of evaluations to reach the hundreds of thousands. At this scale, Optuna’s execution speed and memory consumption become bottlenecks, making efficient search difficult.

Given that various software projects such as Ruff, Polars, and Pydantic V2 have achieved dramatic speedups through Rust in recent years, we developed and released Rustuna, a fast Rust implementation of Optuna. Its main features are as follows:

1. Fast sampler implementations: Rustuna supports single-objective TPE, multi-objective TPE, NSGA-II, CMA-ES, and Random Search. Depending on the sampler used and the number of evaluations, Rustuna achieves speedups ranging from several-fold to several hundred-fold when the objective function evaluation itself is quick.
2. Memory-efficient storage implementation: When running large-scale searches with hundreds of thousands of evaluations, memory consumption can grow to non-negligible levels. Rustuna includes a feature to discard evaluation history no longer needed for the search from storage, avoiding the problem of memory consumption and execution time degrading linearly as the number of evaluations increases.
3. Zero Python runtime dependency: Whereas Optuna depends on packages such as NumPy and SQLAlchemy, Rustuna has no Python dependencies except for some features. This also helps mitigate the risk of supply chain attacks that have become increasingly common in recent years.

For more details on Rustuna, please see the release blog post and official documentation below.

• Release blog post: https://medium.com/optuna/announcing-rustuna-cc82a6815bf7
• GitHub: https://github.com/optuna/rustuna
• Official documentation: https://rustuna.readthedocs.io/

Conclusion

With Optuna v5 and Rustuna, you can now tackle black-box optimization faster and more effectively across domains ranging from machine learning hyperparameter optimization to materials science and structural design. Optuna v5 packs improvements that many users will notice directly: the first default sampler change since v1.5, a refined constrained optimization API, and faster parameter importance calculation. For use cases requiring many evaluations, we also encourage you to try Rustuna.

Our development team will continue to expand functionality and improve performance so that the Optuna family can help solve even more real-world problems. Please try out the latest release and share your feedback through GitHub Issues and Discussions.

Reference

1. Ozaki, Y., Tanigaki, Y., Watanabe, S., Nomura, M., & Onishi, M. (2022). Multiobjective tree-structured parzen estimator. Journal of Artificial Intelligence Research, 73, 1209–1250.
2. Watanabe, S. (2023). Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance. arXiv preprint arXiv:2304.11127.
3. Hutter, F., Hoos, H., & Leyton-Brown, K. (2014, January). An efficient approach for assessing hyperparameter importance. In International Conference on Machine Learning (pp. 754–762). PMLR.
4. Watanabe, S., Bansal, A., & Hutter, F. (2023). PED-ANOVA: Efficiently quantifying hyperparameter importance in arbitrary subspaces. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI ’23) (Article 488, pp. 4389–4396).
5. Baba, K., Ozaki, Y., & Watanabe, S. (2026, August). Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 (pp. 57–68).

Releasing Optuna v5.0, an Open-Source Black-Box Optimization Library was originally published in Optuna on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, we will introduce the parallelization enhancements for GPSampler scheduled for release in Optuna v4.8 and v4.9.

GPSampler is a Bayesian optimization sampler based on Gaussian Processes. It demonstrates particularly high performance in optimizing continuous functions. Since its introduction in Optuna v3.6, it has been extended in various ways, including support for constrained multi-objective optimization.

However, in the implementation of GPSampler up to v4.7, Gaussian process regression was performed using only completed samples. As a result, in batch optimization, the next candidate points were selected from nearly identical regression models based on the same evaluated data. This led to the problem of repeatedly suggesting almost the exact same points.

To resolve this issue, Optuna v4.8 introduces an enhanced parallelization strategy using the Constant Liar approach for unconstrained single-objective optimization. Additionally, Optuna v4.9 will introduce parallelization enhancements using the Kriging Believer approach for constrained and multi-objective optimization. This makes it possible to suggest the next trial points while taking running trials (points currently being evaluated) into account, leading to more diverse and meaningful proposals during batch optimization. These features do not require any special arguments; they are automatically enabled simply by executing parallel optimization as usual.

In this article, we will provide an overview of the newly introduced algorithms, followed by visualization and benchmark results.

Handling Running Trials in GPSampler

For this parallelization enhancement, we implemented methods based on Constant Liar and Kriging Believer [1]. In these methods, dummy objective values are assigned to the running trials, and the regression model is updated as if those points had already been evaluated with those values, as shown in Figure 1. This enables the suggestion of the next evaluation point while taking the running trials into account.

In the Constant Liar approach, a constant dummy value is assigned to the running trials. Specifically, methods to derive this dummy value include using the best, worst, or mean values obtained from the search so far. Using the best value results in more exploitation-oriented behavior, while using the worst value results in more exploration-oriented behavior. As will be discussed later, based on our benchmark results, we have adopted the strategy of using the best value as the dummy value (Constant Liar [best]) for unconstrained single-objective optimization in GPSampler.

In the Kriging Believer approach, the predictive values of the regression model — trained solely on evaluated points — are used as the dummy values for the running trials. The advantage of this method lies in its ease of natural extension to multi-objective and constrained optimization. In those settings, it is difficult to simply define the “current best” or “current worst” as in Constant Liar, making the choice of dummy values non-trivial. On the other hand, since Kriging Believer uses the model’s predictions directly, it can be applied with the same underlying concept even when the number of objective functions or constraints increases. For these reasons, we have adopted Kriging Believer for the parallelization enhancements in multi-objective and constrained optimization scheduled for release in v4.9.

Visualizing Suggested Points in Batch Optimization

First, we visualized the points proposed by GPSampler during batch optimization. Here, we evaluate the sampler on an unconstrained single-objective optimization problem using the Branin-Hoo function [2]. In GPSampler, the first 10 trials are selected randomly. After that, we proposed a total of 50 trials with a batch size of 5. The points proposed during these 50 trials are shown as red dots in Figure 2.

As shown in Figure 2(a), when parallelization strategies like Constant Liar are not used (i.e., v4.7 and earlier), almost identical points were repeatedly proposed. As a result, batch optimization sometimes evaluated almost the same candidates multiple times.

On the other hand, Figure 2(b) shows the distribution of proposed points when using Constant Liar [best], introduced in v4.8. As the figure illustrates, by taking running trials into account through the newly introduced enhancement, the repeated proposal of identical points was significantly reduced. As a result, more diverse candidate points are selected in each batch, leading to improved optimization performance.

Performance Benchmarking

Next, we conducted a benchmark to quantitatively evaluate optimization performance. We compared the following 5 strategies:

• Ignore running trials: Implementation up to v4.7 that does not account for running trials.
• CL[best]: Fill running trials with the best value among the evaluated points.
• CL[worst]: Fill running trials with the worst value among the evaluated points.
• CL[mean]: Fill running trials with the mean value among the evaluated points.
• KB: Fill running trials with the predictive values of the regression model.

Experimental Setup

• Benchmark Problems: Comparisons were made across 8 types of unconstrained single-objective optimization problems:
  - BBOB problems: 5 types (function_id=1,6,10,15,20)
  - HPO (Hyperparameter Optimization) problems: 3 types (dataset_id=0,1,2)
• Evaluation Parameters: For each of the 8 problems above, we prepared 12 setting patterns by combining the following two factors:
  - Number of trials (n_trials): [25, 50, 75, 100]
  - Batch size (batch_size): [5, 10, 50]

Evaluation Metrics and Statistical Methods

To ensure a highly accurate comparison, we evaluated each method using the following steps:

1. Repeated Trials: For each setting, we executed 100 runs using different random seeds and obtained the best value for each run.
2. Statistical Testing: We performed a one-sided Mann–Whitney U test (significance level: alpha = 0.05) between the methods.
3. Win Count Aggregation: We counted the number of times a method was judged to be “significantly superior” compared to the other methods, recording this as its “win count.”

Computing Environment: The benchmark was conducted on an Ubuntu 24.04 environment with an x86_64 architecture. We used an Intel 12th Gen Core i7–1260P CPU (12 cores, 16 threads, max 4.70 GHz).

The results are as follows:

Compared to the conventional strategy that ignores running trials, methods that factor them in showed significantly better results across many problem settings. Among them, CL[best] achieved the highest win count, proving to be particularly effective in this benchmark. We believe this is because many of the problems used here have relatively smooth objective functions with globally simple structures, where promising candidates were likely to exist around regions that already showed good values. Based on these results, Optuna has officially adopted CL[best] for unconstrained single-objective optimization.

On the other hand, there were a few cases where the conventional implementation showed significantly better results in certain settings. This occurred in problem settings using the Sphere Function [3] from BBOB. The Sphere Function is an extremely simple problem where the objective function is the square of the Euclidean distance from the optimal solution. Therefore, it was possible to reach the vicinity of the optimal solution in the early stages, and the subsequent behavior of repeatedly exploring the immediate surrounding area likely worked to its advantage.

Conclusion

In Optuna v4.8 and v4.9, we improved the parallelization of GPSampler by introducing Constant Liar and Kriging Believer. These improvements prevent the same point from being suggested repeatedly in batch optimization, and we confirmed that they significantly improve optimization performance across many benchmarks.

Going forward, we plan to further improve parallelization strategies in GPSampler, including the introduction of Monte Carlo-based acquisition functions. Stay tuned!

References

[1] Ginsbourger, D., Le Riche, R., & Carraro, L. (2010). Kriging is well-suited to parallelize optimization. In Computational intelligence in expensive optimization problems (pp. 131–162). Berlin, Heidelberg: Springer Berlin Heidelberg.

[2] Branin, F. H. (1972). Widely convergent method for finding multiple solutions of simultaneous nonlinear equations. IBM Journal of Research and Development, 16(5), 504–522.

[3] The blackbox optimization benchmarking (bbob) test suite (https://hub.optuna.org/benchmarks/bbob/)

Improving Optuna’s GPSampler Parallelization by Considering Running Trials was originally published in Optuna on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Optuna Dashboard (GitHub: https://github.com/optuna/optuna-dashboard) is a web-based dashboard that makes it easy to review Optuna’s optimization history. As introduced in the Optuna v5 roadmap, Optuna is currently enhancing its integration with large language models (LLMs) for its next major release. Introducing the LLM integration in Optuna Dashboard is one of its key milestones.

Optuna Dashboard v0.20.0 is the first version to support LLM integration. This article provides an overview of the newly introduced LLM integration features, along with details about the design and implementation considerations behind them.

Overview of LLM Integration

Enabling the LLM integration features currently makes the following two functionalities available:

1. Natural language-based Trial filtering
2. Automatic chart generation

The natural language-based Trial filtering function, illustrated in Figure 1, allows users to refine which Trials appear in graphs and tables using natural language. The traditional Optuna Dashboard only supported limited filter conditions. This is because implementing comprehensive filter options would significantly complicate the UI, making it less user-friendly. This new feature enables flexible filtering through simple UI by having the filter functions generated by LLM based on user instructions.

The next feature is the automatic chart generation feature. Optuna Dashboard has supported generic charts useful for various optimization tasks, including visualizations of evaluation metric trends, Pareto front solutions, and contour plots. However, supporting visualization features specific to certain optimization tasks — which might be needed only for particular use cases — had been challenging. With the automatic chart generation feature, as shown in Figure 2, users can request specific visualizations, and the LLM will generate visualization functions using Plotly.js, thereby enabling visualizations that were previously unsupported for task-specific use cases.

For detailed instructions on using the LLM integration feature, please refer to the official documentation tutorial: Tutorial: LLM Integration — Optuna Dashboard Documentation.

The Internal Execution Flow

The two features mentioned earlier are very similar in terms of design and implementation. Visualizing their internal execution flow in a sequence diagram yields Figure 3 below.

The system operates as follows: When a user inputs and submits instructions, the API server receives these texts. The server then generates a prompt based on a predefined template and queries the LLM Provider (such as the OpenAI API). The LLM Provider returns a string containing a JavaScript function that can be executed via eval(). The JavaScript function string is then displayed to the user for confirmation. If no issues are found, eval() is called, and the evaluation results are sent to the parent page via postMessage() for display.

The prompt template varies depending on the functionality but generally consists of four core elements:

1. Filtering function generation instructions (e.g., “Please create a filtering function for …”)
2. TypeScript type definitions (e.g., type Trial = {trial_id: number; … })
3. Few-shot examples (see the source code for details)
4. (Only during retries) The function definition from previous evaluation attempts along with its error message

For clarity in Figure 3, we omitted the details of retrying failed evaluations. When an LLM-generated JavaScript function evaluation fails, the system embeds the attempted function and error message back into the prompt template for retry. Many successful evaluations occur through retries, making this a practical feature.

When examining the overall execution flow, there are no particularly specialized aspects from a design perspective. However, a naive implementation would involve executing functions generated by the LLM via eval(), which raises certain risks and concerns. To address this, Optuna Dashboard incorporates several safeguards. Below, we’ll discuss these implementation-specific design considerations.

Confirmation Dialog

By design, Optuna Dashboard is not intended for public sharing among unrestricted users. Instead, it is meant to be used locally on individual computers or within trusted internal networks among limited members. Even so, fully trusting code generated by LLMs remains challenging, as there remains a risk of unintended behavior occurring.

One common mitigation approach is confirmation dialogs, similar to those implemented by Claude Desktop and various coding agents. Optuna Dashboard incorporates a confirmation dialog as shown in Figure 4, allowing users to examine the contents of JavaScript functions before execution and choose whether to proceed.

The execution confirmation dialog includes a convenient “Never Ask Again” button. Upon clicking this button, the information is stored in the browser’s sessionStorage, preventing the dialog from appearing again in subsequent sessions. Unlike localStorage, sessionStorage is tied to individual browser tabs — allowing permission in one tab does not carry over to others, and the data is cleared when the tab is closed (see: Web Storage API).

Evaluating LLM-generated code inside the iframe-sandboxed environment

Not all Optuna Dashboard users will be proficient in JavaScript, and visual inspection alone has inherent limitations. To complement the confirmation dialog, we implemented eval() execution within a sandboxed environment using <iframe sandbox=”allow-scripts”></iframe> tags.

As noted in MDN’s documentation, eval() can execute arbitrary JavaScript code, making it potentially dangerous if malicious code is passed to it. If such code were executed, it could read application variables or directly modify the DOM. Using an iframe tag creates a separate execution context that is isolated from the parent page, preventing direct access to the parent page’s DOM or JavaScript variables. Furthermore, by restricting all attributes except “allow-scripts” in the sandbox attribute, we can prevent issues like popup displays, form submissions, and access to localStorage/sessionStorage.

To illustrate this implementation in detail, we present a simplified excerpt from Optuna Dashboard’s source code below (the full source code is available here). We embed an iframe tag as an invisible element on the page, sending the JavaScript function (as a string) to be evaluated along with the required parameters via postMessage(), and then return only the result of the eval() execution back to the parent page via postMessage().

const iframeSrcDoc = `
    <script>
      window.fetch = () => { throw new Error("Unexpected") }
      window.XMLHttpRequest = () => { throw new Error("Unexpected") }
      window.addEventListener('message', (event) => {
        const { data } = event;
        if (typeof data !== 'object' || data.type !== 'filter') return;
        const { trials, filterFuncStr } = data;
        try {
          const filterFunc = eval('(' + filterFuncStr + ')');
          const result = trials.filter(filterFunc);
          parent.postMessage({ type: 'result', filteredTrials: result }, '*');
        } catch (e) {
          parent.postMessage({ type: 'result', filteredTrials: [], error: String(e) }, '*');
        }
      });
    </script>
  `
return (
  <iframe
    ref={iframeRef}
    sandbox="allow-scripts"
    style={{ display: "none" }}
    srcDoc={iframeSrcDoc}
  />
)

Important note: The sandboxed environment provided by iframe tags does not eliminate all security concerns. For example, iframe tags alone cannot restrict external network communications using the fetch API, requiring additional measures like Content Security Policy (CSP) to control network access.

Conclusion

This article introduced the overview and internal design/implementation details of our LLM integration feature. The LLM-based trial filtering enables flexible filtering through an extremely simple UI, while the automatic generation of Plotly charts based on user requests would have been nearly impossible to achieve without LLM integration. Our LLM integration features still have significant room for further development, and we plan to continue refining them in preparation for Optuna v5.

If you’ve found this article interesting and would like to try out the LLM integration functionality, please refer to the tutorial and we’d greatly appreciate your feedback.

Optuna Dashboard Integrates with LLM: Enabling Data Filtering and Graph Generation using Natural… was originally published in Optuna on Medium, where people are continuing the conversation by highlighting and responding to this story.

We are excited to announce the release of Optuna 4.6, the latest version of our black-box optimization framework. This release significantly improves the user experience and optimization performance, with a particular focus on the following highlights:

• Introduction of an interactive interface powered by Large Language Models (LLMs)
• Further speed enhancements for GPSampler
• Full support for multi-objective and constrained optimization in AutoSampler
• Additions of robust Bayesian optimization packages

Beyond these significant improvements, this version also includes several updates to enhance usability, including an OpenTelemetry integration tutorial, LaTeX support in OptunaHub, and structured output support for the Optuna MCP Server. In this article, we will detail these noteworthy updates.

Get started with the new features by installing the latest version with the following command!

pip install -U optuna optuna-dashboard optunahub

Optuna Dashboard LLM Integration

We are introducing an LLM integration feature to the Optuna Dashboard. By enabling this, you can now utilize two key functions: 1) filtering trials using natural language and 2) automatic graph generation.

The natural language trial filtering feature, as shown in Figure 1, allows you to filter the trials displayed in graphs and tables using natural language queries. Previously, the filter conditions in Optuna Dashboard were quite limited, as supporting a wide variety of conditions would complicate the UI and hinder usability. With this new feature, the LLM generates a filter function based on your instructions, enabling flexible filtering through a simple interface.

Next, let’s take a look at the automatic graph generation feature. Optuna Dashboard supports general-purpose graphs for any optimization task, including visualizing objective value history, Pareto fronts, and contour plots. However, it was challenging to support visualizations tailored to specific optimization tasks that are needed in only a few use cases. Now, as shown in Figure 2, you can describe the plot you want in natural language, and the LLM will generate a visualization function using Plotly.js. This enables task-specific visualizations that were previously unsupported.

For instructions on how to use the LLM integration, please refer to the Tutorial: LLM Integration — Optuna Dashboard documentation. Please note that the performance of these features depends on the LLM being used.

Speeding up GPSampler

GPSampler is a Gaussian Process-based Bayesian optimization sampler known for its high sample efficiency. However, when the evaluation time of the objective function is less than a few hours, the sampling time of GPSampler can become a bottleneck.

In Optuna v4.6, we have made several modifications to GPSampler’s internal implementation to accelerate sampling time while maintaining optimization performance. Specifically, we implemented caching for repeated calculations, removed unnecessary operation calls for empty tensors, and parallelized the optimization of the acquisition function. In the experimental setup shown in Figure 3, we achieved a speedup of nearly 8x. The parallelization of the acquisition function optimization is detailed in a separate article (in Japanese). Please note that the actual speedup may vary depending on the computational environment.

Full Support for Multi-objective and Constrained Optimization in AutoSampler

AutoSampler is the most popular package in OptunaHub, enabling efficient optimization by automatically selecting the best algorithm. In Optuna v4.6, we have fully implemented sampler selection rules for multi-objective and constrained optimization (Figure 4). The latest AutoSampler now strategically uses samplers like GPSampler for constrained and multi-objective optimization (implemented across Optuna v4.2-v4.5) and NSGAIIISampler for many-objective optimization, leading to significant improvements in benchmark performance.

For more details, please see our blog post, “AutoSampler: Full Support for Multi-Objective & Constrained Optimization.”

Additions of Robust Bayesian Optimization Packages

Robust Bayesian optimization is a method for ensuring that optimal solutions obtained through simulations can be stably reproduced in real-world experiments, even when parameter values are noisy. In practice, maintaining experimental parameters (e.g., temperature and pressure) precisely is challenging and can lead to variations in results. To address this issue, research and development are underway to develop a framework for finding optimal solutions in which the objective function’s value does not degrade significantly even when the set values deviate slightly.

In Optuna v4.6, we have made CARBO and Value at Risk (VaR), which are robust Bayesian optimization methods developed in the fields of experimental chemistry and machine learning, respectively, available via OptunaHub. As shown in Figure 5, the introduced methods can perform robust optimization even in the presence of input noise.

Other Updates

Tutorial for Monitoring Optuna Dashboard with OpenTelemetry

We have released a new tutorial, “Tutorial: Integrating with OpenTelemetry,” for monitoring Optuna Dashboard using OpenTelemetry. OpenTelemetry is a framework that simplifies the monitoring and tracking of performance and resource usage in applications and systems. By integrating Optuna Dashboard with OpenTelemetry (Figure 6), you can:

• Monitor HTTP request metrics (latency, error rate, throughput).
• Track the performance of database queries.
• Export metrics to Prometheus for storage and visualization in Grafana.

If you are interested in monitoring your Optuna Dashboard, we encourage you to try it out.

LaTeX Support on OptunaHub Package Detail Pages

We’ve added support for LaTeX mathematical expressions on the package detail pages in OptunaHub. With this update, package developers can now use both inline ($expr$, `$expr$`) and block ($$expr$$) expressions with GitHub-compatible syntax. For users, this dramatically improves documentation readability, as formulas are now beautifully rendered. Figure 7 shows an example of formulas from the CARBOSampler’s package detail page.

Optuna MCP Server Now Supports Structured Output

Optuna MCP Server now supports Structured Output, a feature introduced in the latest Model Context Protocol specification (2025–06–18). This enhancement improves stability by automatically validating the results returned from the Optuna MCP Server to the MCP client against an output schema generated from type annotations. For more technical details, please refer to the official documentation.

What’s Ahead

In this article, we have introduced the release highlights of Optuna v4.6. These highlights are directly tied to the three core concepts of the Optuna v5 roadmap, which we unveiled this past May.

Unleashing the Potential of Black-box Optimization with Generative AI

• Introduction of Optuna Dashboard LLM Integration
• Structured Output Support for Optuna MCP Server

Making Optuna Available Anytime, Anywhere

• Tutorial for Monitoring Optuna Dashboard with OpenTelemetry

Continuing the Sustainable Growth of Optuna

• Speeding up GPSampler
• Full Support for Multi-Objective and Constrained Optimization in AutoSampler
• Additions of Robust Bayesian Optimization Packages
• LaTeX Support on OptunaHub Package Detail Pages

We are preparing to change the default sampler, prototyping performance improvements by replacing components with Rust, and validating prompt-optimization techniques. We plan to continue releasing exciting new features as we move toward Optuna v5, so please stay tuned!

Contributors

The v4.6 release includes contributions from the following contributors. Thank you all!

@AddyM, @GabrielRomaoG, @Jongwan93, @Kaichi-Irie, @ParagEkbote, @Zrahay, @c-bata, @contramundum53, @dross20, @euangoodbrand, @fusawa-yugo, @gen740, @jaikumarm, @kAIto47802, @ktns, @nabenabe0928, @nihalsiddiqui7, @not522, @satyarth7srivastava, @sawa3030, @toshihikoyanase, @unKnownNG, @y0z

Announcing Optuna 4.6 was originally published in Optuna on Medium, where people are continuing the conversation by highlighting and responding to this story.

AutoSampler is an intelligent sampler that automatically selects an optimization algorithm implemented in Optuna for a given problem, enabling efficient optimization. As the most popular package on OptunaHub, it receives over 30,000 weekly downloads. The initial version of this feature, as stated in a blog post published shortly after its release, implemented special automatic selection rules for single-objective optimization. However, for multi-objective optimization and constrained optimization, it defaulted to Optuna’s standard sampler, with full support for these problem settings designated as future work.

With the development of Optuna v4.6 (to be released next month), we have enhanced AutoSampler to fully support multi-objective and constrained optimization — a long-requested feature. This blog post will introduce these new enhancements, explain how to use them, and present benchmark results demonstrating their performance. (Note: The new AutoSampler is available now.)

Enhanced Support for Multi-Objective & Constrained Optimization

The new AutoSampler builds upon its original design philosophy by automatically selecting the most suitable sampler for a given optimization problem. It analyzes factors such as the evaluation budget, search space, constraints, and the number of objectives to choose a sampler that empirically delivers results equal to or better than the default.

With this update, the selection rules now fully support multi-objective and constrained optimization. The logic automatically switches between several powerful samplers (see Figure 1):

• TPESampler: Best for complex search spaces involving categorical variables and conditional branching.
• GPSampler: Ideal for continuous and integer search spaces, now with full multi-objective and constraint support.
• NSGAIISampler & NSGAIIISampler: Scalable choices for problems with a high number of evaluations (NSGA-II) or many objectives (NSGA-III).

This significant enhancement was made possible by the full implementation of constraint and multi-objective support for GPSampler in Optuna v4.2-v4.5.

How to Use AutoSampler

Getting started with the latest AutoSampler is straightforward. Here’s how you can set it up and integrate it into your Optuna study.

First, make sure you have the necessary packages installed. You’ll need optuna version 4.5 or higher. You can install optuna, optunahub, and the AutoSampler’s dependencies with the following commands:

pip install -U optuna optunahub
pip install -r https://hub.optuna.org/samplers/auto_sampler/requirements.txt

Once the packages are installed, you simply load the "samplers/auto_sampler” package using the optunahub.load_module function, instantiate the AutoSampler included in the package, and use it just like any other Optuna sampler. Here is a runnable example for a multi-objective optimization problem:

import optuna
import optunahub


def objective(trial: optuna.Trial) -> tuple[float]:
    x = trial.suggest_float("x", 0, 5)
    y = trial.suggest_float("y", 0, 3)
    return  4 * x**2 + 4 * y**2, (x - 5)**2 + (y - 5)**2

study = optuna.create_study(
    directions=["minimize", "minimize"],
    sampler=optunahub.load_module("samplers/auto_sampler").AutoSampler()
)
study.optimize(objective, n_trials=300)

print(study.best_trials)

If you’ve used AutoSampler before, your locally cached version might be outdated. To ensure you have the latest version, run load_module once with the force_reload=True flag.

python
>>> import optunahub; optunahub.load_module("samplers/auto_sampler", force_reload=True)

Performance Evaluation

We evaluated the latest AutoSampler against Optuna’s default sampler in both multi-objective and constrained optimization scenarios. The results highlight the benefits of its intelligent, adaptive approach.

Multi-Objective Optimization

For our multi-objective benchmark, we used the widely recognized WFG1 problem. As shown in Figure 2, the AutoSampler demonstrates superior search performance. Unlike the default sampler, which always uses the NSGA-II algorithm, AutoSampler dynamically switches between four powerful methods based on the problem’s characteristics (e.g., number of evaluations, objectives, and search space): GPSampler, TPESampler, NSGAIISampler, and NSGAIIISampler. This automatic selection allows it to find better solutions by adapting its strategy to the task at hand.

Constrained Optimization

We also compared performance on constrained problems using the 5-dimensional Rotated Rastrigin function with three constraints in the bbob-constrained benchmark set. As seen in Figure 3, the AutoSampler again outperforms the default.

Its advantage comes from new selection rules that incorporate the powerful, constraint-aware GPSampler (introduced in Optuna v4.2). This enables it to navigate constrained search spaces much more effectively. Furthermore, for problems that are both constrained and multi-objective, it leverages the even newer GPSampler enhancements from Optuna v4.5.

Wrapping Up

In this update, we’ve unveiled a significantly enhanced AutoSampler with full support for multi-objective and constrained optimization. By integrating AutoSampler into your workflow, you can achieve performance that is empirically equal to or better than the default sampler, all without the complex task of manually selecting the right algorithm. This powerful convenience allows you to focus on your problem while Optuna handles the optimization strategy.

This release fulfills a key milestone on our Optuna v5 roadmap: “Feature upgrades for AutoSampler to enable multi-objective optimization and constrained optimization with automatic algorithm selection.” We are continuing to develop new features and enhance existing ones as we move closer to the official v5 release. We encourage you to take advantage of these new capabilities and stay tuned for more updates to come!

AutoSampler: Full Support for Multi-Objective & Constrained Optimization was originally published in Optuna on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

This article introduces the constrained multi-objective optimization feature of GPSampler released in Optuna v4.5.

Gaussian process-based Bayesian optimization (GPBO) has gained popularity in various domains, including materials science and machine learning. The initial release of GPSampler aimed to provide a simple and user-friendly implementation of GPBO. While its component customization, such as kernels and acquisition functions, is less flexible than that of BoTorch and Ax (maintained by Meta Platforms, Inc.), GPSampler runs faster with fewer dependencies.

At the same time, the first version of GPSampler supported only limited problem setups, motivating further development. Optuna v4.2 extended GPSampler to handle inequality constraints, as announced in this article. Optuna v4.4 then supported the multi-objective optimization. With the latest release, Optuna v4.5 now supports constrained multi-objective optimization.

The new feature is useful in many domains, including materials science, machine learning, and drug discovery, where multiple objectives must be balanced under realistic physical or experimental constraints. Our constrained multi-objective GPBO in Optuna v4.5 directly offers this capability, with the potential to accelerate scientific progress across a wide range of domains.

In this article, we explain its usage, provide a brief technical overview, and present benchmarking results against samplers available in Optuna.

How to Use GPSampler in Optuna

GPSampler requires additional dependencies on top of optuna:

$ pip install optuna==4.5.0

# scipy and torch are also necessary for GPSampler.
$ pip install scipy
$ pip install torch --index-url https://download.pytorch.org/whl/cpu

Once the installation is done, you can easily use GPSampler by passing it to your study:

import optuna


def objective(trial: optuna.Trial) -> tuple[float, float]:
    x = trial.suggest_float("x", -5.0, 5.0)
    y = trial.suggest_float("y", -5.0, 5.0)
    c = x**2 + y**2 - 4.0
    trial.set_user_attr("c", c)
    return x**2 + y**2, (x - 2) ** 2 + (y - 2) ** 2


def constraints(trial: optuna.trial.FrozenTrial) -> tuple[float]:
    c = trial.user_attrs["c"]
    return (c,)


sampler = optuna.samplers.GPSampler(constraints_func=constraints)
study = optuna.create_study(sampler=sampler, directions=["minimize"] * 2)
study.optimize(objective, n_trials=100)

Multi-Objective Optimization with GPSampler

The acquisition function of our implementation is the logarithm of the Expected Hypervolume Improvement (log EHVI), which essentially relies on Eq. (1) of Daulton et al. [1] and Algorithm 2 of Lacour et al [6]. Technical details are available in the following PRs:

• #6039: Implement the box decomposition algorithm to speed up the hypervolume improvement calculations.
• #6052: Implement the quasi Monte-Carlo-based log EHVI.
• #6069: Add support for multi-objective optimization in GPSampler.

As discussed in the PR, the box decomposition algorithm improves the time complexity of log EHVI, resulting in a 6–800 times speedup over the naive approach and making our implementation practical up to four-objective problems. Note that although our implementation mostly aligns with the BoTorch implementation, some details rely on different logic. For further information, please refer to this PR.

Constrained Multi-Objective Optimization with GPSampler

On top of the multi-objective optimization described above, we extend GPSampler to handle inequality constraints following the approaches of J. Gardner et al. [3] and M. Gelbart et al. [4]. In this setting, GP models are employed for both the objective functions and the constraint functions, with correlations between them being ignored. See the following PRs for more details:

• #6198: Implement the constrained log EHVI.
• #6224: Add support for constrained multi-objective optimization in GPSampler.

In our implementation, the feasibility of a candidate point is evaluated using the Probability of Improvement (PI). The overall acquisition function is constructed as the product of the EHVI and the feasibility probabilities derived from PI.

Performance Benchmarking

We conduct benchmarking to see the effectiveness of constrained multi-objective optimization by GPSampler. Specifically, we compare this new feature against: (i) multi-objective GPSampler without constraint handling (hereafter referred to as unconstrained GPSampler) to assess the benefit of incorporating constraints; and (ii) TPESampler and NSGAIISampler in Optuna, that support constrained multi-objective optimization.

Benchmarking Setup

Benchmarking Problem. We use the C2-DTLZ2 problem [5], newly added to OptunaHub for constrained multi-objective optimization benchmarking. See the OptunaHub documentation and the corresponding PR for more details, and refer to the original papers [2, 6] for the detailed definitions of the objective and constraint functions. We set the number of objectives to two and the number of variables to three.

Implementation Details. We run each optimization for 300 trials and repeat it five times with different random seeds to evaluate the standard error. Since the C2-DTLZ2 problem is deterministic, i.e., no noise is included in the objective functions, we use deterministic_objective=True for GPSampler. See this PR for the implementation we used.

Machine Specifications. The benchmarks are conducted on a computer running Arch Linux with Intel® Core™ i9–14900HX processor (24 cores, 32 threads, up to 5.8GHz) and Python 3.11.0.

Results and Discussion

Analysis of Obtained Pareto Fronts and the Effects of Constraint Consideration

Figure 1 illustrates the obtained Pareto fronts to highlight the distribution and quality of the solutions.

Effectiveness of Constraint Handling in GPSampler

The C2-DTLZ2 problem directly introduces infeasible regions on the Pareto front to evaluate the algorithm’s ability to handle disconnected Pareto fronts [5]. As a result, while it is possible to obtain Pareto solutions without considering the constraints, such attempts often include wasted evaluations in infeasible regions, making constrained optimization particularly important.

Comparing the constrained GPSampler (Figure 1(a)) with the unconstrained GPSampler (Figure 1(b)), the former yields fewer infeasible observations (colored gray) between feasible Pareto solutions (colored blue), showing that the constrained one successfully captures the constraints and reduces wasted evaluations.

Comparison with Other Samplers

Compared to TPESampler (Figure 1(c)), GPSampler (Figure 1(a)) also generates fewer wasted evaluations in infeasible regions. At the same time, the Pareto front close to where one of the objective values approaches zero is more thoroughly covered by TPESampler. This is because, owing to the nature of GP-based acquisition functions, the optimization tends to oversample near the boundaries, which reduces variance in those regions and makes further exploration of the adjacent Pareto front more difficult.

Furthermore, as shown in Figure 1(d), NSGAIISampler has not yet converged after 300 trials, with few optimal solutions achieved, highlighting the superior performance of GPSampler.

Analysis of Feasible Hypervolume History

Figure 2 shows the feasible hypervolume history, enabling us to assess how quickly and effectively each sampler improves performance under the given constraints.

As shown in Figure 2, the hypervolume history of GPSampler reaches the final performance of TPESampler with fewer than 50 trials and surpasses NSGAIISampler much earlier, demonstrating its superior sample efficiency. The rapid convergence to the best hypervolume is particularly important for computationally expensive objectives, since reducing function evaluations directly leads to substantial overall time savings.

Conclusion

We extended GPSampler to support constrained multi-objective optimization in Optuna v4.5. Benchmarks on the C2-DTLZ2 problem [5] confirmed that the new feature reduces wasted evaluations in infeasible regions and achieves faster convergence in feasible hypervolume compared to TPESampler and NSGAIISampler. This new functionality marked a major step forward for Optuna, making it a strong candidate for black-box optimization of computationally expensive objectives. Please give it a try!

References

[1] Daulton, S., Balandat, M., and Bakshy, E. Differentiable expected hypervolume improvement for parallel multi-objective Bayesian optimization. In Advances in Neural Information Processing Systems, volume 33, pp. 9851–9864, 2020.

[2] Deb, K., Thiele, L., Laumanns, M., and Zitzler, E. Scalable test problems for evolutionary multi-objective optimization. In Evolutionary Multi-objective Optimization, pp. 105–145. Springer-Verlag, 2005.

[3] Gardner, J., Kusner, M., Zhixiang, W., Weinberger, K., and Cunningham, J. Bayesian Optimization with Inequality Constraints. In Proceedings of the 31st International Conference on Machine Learning, volume 32, pp. 937–945. PMLR, 2014.

[4] Gelbart, M. A., Snoek, J., and Adams, R. P. In Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence, pp. 250–259. AUAI Press, 2014.

[5] Jain, H., and Deb, K. An Evolutionary Many-Objective Optimization Algorithm Using Reference-Point Based Nondominated Sorting Approach, Part II: Handling Constraints and Extending to an Adaptive Approach. In IEEE Transactions on Evolutionary Computation, 18(4):602–622, 2014.

[6] Lacour, R., Klamroth, K., and Fonseca, CM. A box decomposition algorithm to compute the hypervolume indicator. Computers & Operations Research, volume 79, pp. 347–360, 2017.

[Optuna v4.5] Constrained Multi-Objective Gaussian Process-Based Sampler (GPSampler) was originally published in Optuna on Medium, where people are continuing the conversation by highlighting and responding to this story.