Abstract: Tabular data, spreadsheets organized in rows and columns, are ubiquitous across healthcare, business and finance. The fundamental prediction task of filling in missing values of a label column based on the rest of the columns is essential for thousands of use cases of high societal and commercial value, from early cancer detection to predictive maintenance and demand forecasting. While gradient-boosted decision trees have dominated tabular data for two decades, we demonstrate that this is rapidly changing, with the foundation model revolution having arrived at tabular data. We will show the methods behind this and their extensions to causality, interpretability and robustness, and demo various agentic extensions that achieve the dream of AutoML to democratize machine learning, allowing even non-experts to use state-of-the-art machine learning without writing a single line of code.

Abstract:  Many problems in industry involve objectives that are difficult to write down explicitly: the quality of a generative model's outputs, the relevance of a recommender, the feel of a user interface, or the desirability of a candidate molecule. In these settings, a decision-maker, whether a human expert or another AI system, may struggle to assign numerical scores but can still compare alternatives, for example by choosing the best option from a set or ranking several options from best to worst.

Preferential Bayesian optimization (PBO) studies how to use this kind of comparative feedback for sequential decision-making. Instead of observing function values directly, we place a Gaussian process prior over a latent utility function and update it using preference data. This changes the modeling problem: the surrogate, the likelihood, and the acquisition strategy must all be adapted to observations that take the form of comparisons. The resulting methods retain several features that make Bayesian optimization useful for expensive optimization problems, including sample efficiency and uncertainty-aware query selection.

This tutorial will introduce the main ideas behind PBO and survey how the field has developed. We will discuss how comparison-based feedback changes the usual Bayesian optimization loop, and how these ideas connect to reward modeling and RLHF through shared preference-modeling tools. We will then consider what changes when feedback is expressed in natural language. In that context, we will cover LILO, Language-in-the-Loop Optimization, which uses an LLM as an interface between free-form feedback and structured preference data for a Gaussian process model. The aim is to examine how language can help express richer feedback and prior knowledge while leaving the optimization problem to the Bayesian optimization model.

By the end of the tutorial, attendees should have a clear map of the PBO landscape, a sense of where language models may be useful in this setting and where their role is limited, and a starting point for exploring open questions and available tools.

Abstract: Deploying neural networks on resource-constrained edge devices requires balancing predictive performance with latency, memory, and energy consumption. Neural Architecture Search (NAS) is a promising method towards automating this complex, hardware-dependent design process. In this talk, I will present our recent work on evolutionary NAS for Edge AI, spanning efficient search strategies, architecture analysis, and deployment on microcontrollers. I will discuss how population-based guidance and similarity-aware exploration can reduce search costs while improving robustness and generalization across tasks. I will then introduce hardware-in-the-loop approaches that close the gap between simulated efficiency and real-world performance. Our end-to-end co-design pipelines produce models that are not only accurate but genuinely executable and efficient at the edge.

Lecture: Agentic AutoML systems

AutoML is undergoing a fundamental paradigm shift to autonomous, LLM-driven agents capable of end-to-end data science reasoning—from exploratory data analysis and open-ended feature creation to self-healing code execution and model iteration. However, building reliable agentic AutoML systems introduces critical design challenges: pure freeform coding agents frequently succumb to context drift and hallucinated metrics, while strictly deterministic task orchestrators strip away the semantic problem-solving power of modern language models. In this talk, we explore the architectures and open-science protocols driving state-of-the-art Agentic AutoML. We also highlight the role of standardized data-grounding frameworks like Eclair, which combine the Model Context Protocol (MCP) with machine-readable metadata standards like Croissant. By providing agents with deterministic tools to discover, preview, and ingest global datasets with rich metadata on the fly, these protocols transform agents from closed script-writers into proactive data science collaborators capable of dynamic data augmentation and autonomous benchmarking.

Lecture: The Science of Benchmarking

In this lecture we explores the evolving science of AI evaluation, charting the methodological shift from fixed, task-specific benchmarks like ImageNet to the complex landscape of modern generative AI. While traditional evaluation frameworks relied on isolated holdout sets—where relative model rankings proved remarkably robust despite dataset reuse—the rise of web-scale generative models has triggered a benchmarking crisis marked by rapid saturation, data contamination, shortcut learning, and rank-aggregation anomalies. This challenge directly impacts AutoML since it fundamentally relies on a clear, accurate evaluation signal to determine which configurations perform best; while traditional AutoML could optimize straightforward metrics like classification accuracy, doing so with LLMs is far more complicated due to subtle cues and benchmark shortcuts. The talk details how scaling laws and post-training alignment influence benchmark outcomes, often incentivizing models to exploit specific test formats rather than cultivate genuine reasoning, and introduces protocols to ensure fairer evaluations. We tackle the fundamental but unsolved question of construct validity, to guard against anthropomorphism and design rigorous control experiments to determine whether AI systems truly grasp underlying concepts or merely mimic performance.

Abstract: Modern AutoML can, for many purposes, be treated as a dependable engine: given data and a task, it will search for a good model. In industrial practice, though, most of the work of building a useful system lies around that search rather than inside it. Drawing on our experience developing an AutoML system at Beckhoff Automation, this talk takes the search itself as a black box and aims less to present polished solutions than to raise awareness of the problems that surround it once the target is a machine on a factory floor.

A first cluster of challenges comes from the deployment target and the physical world it lives in. Models run inside real-time control systems, where inference latency is a hard constraint whose budget varies enormously from one application to the next. Assumptions that hold on academic image benchmarks -- fixed resolutions, clean and unobstructed views -- often break on the shop floor, where cameras might sit behind cages and safety glass and see parts at awkward scales and aspect ratios. The data is not stationary either: lighting, wear, fouling, and changing materials cause genuine drift over time. And models frequently have to be deployed onto machines that are already running, with little room for downtime.

A second cluster arises from who uses the system and how. Our users are, overwhelmingly, domain experts rather than data scientists, and for those, adjusting the dataset is the option to influence the model quality -- every other decision is abstracted away on purpose. This makes data curation a first-class and surprisingly subtle problem: repeatedly revising the dataset in response to reported scores can quietly overfit the test set, and well-meant user edits can leak information across the train/test split. It also makes communication hard in its own right, since "domain expert" covers a wide spectrum of backgrounds: results have to be mapped into each user's own vocabulary, and the small minority of sophisticated users -- some with a genuine data-science background -- will want more options for interaction themselves, which raises the question of how many levels of engagement, up to an expert mode, a single system should offer.

A third cluster concerns everything required for a trained model to be trusted and to run. Latency must be estimated before deployment rather than measured after it; behaviour must be explained to non-experts in terms they find meaningful; the model must be turned into control code that is both efficient and reliable; and, once deployed, it must be monitored and maintained as the world drifts.

What makes all of these challenges compelling is that none of them are afterthoughts to the search, and yet few are truly settled: most are close relatives of questions this community already studies, reshaped by constraints that benchmarks usually abstract away. I hope to leave the audience with a clearer picture of where these problems lie -- and where academic AutoML research could push the solutions that industrial systems already depend on further than current practice.

Abstract: Neural Architecture Search (NAS) automates the design of neural networks, replacing manual, trial-and-error architecture engineering with systematic search over a defined space of candidate networks. A key challenge in NAS is evaluating how well each candidate will perform, since training every architecture to convergence is prohibitively expensive. Performance prediction methods address this by estimating architecture quality more cheaply, and zero-cost proxies have become a particularly popular choice: computed from a single forward/backward pass, they require no training at all. However, they lack solid theoretical grounding — it's unclear why they correlate with validation accuracy, and they show inconsistent behavior across search spaces and tasks. Some are even biased toward superficial properties like operation count. This tutorial introduces the core building blocks of NAS — search spaces, search strategies, and performance estimation — before focusing on performance predictors and zero-cost proxies in depth, covering both established convolutional spaces and newer transformer-based ones. 

Abstract: This talk introduces BoFire, an open-source Python package that combines Bayesian Optimization (BO) with classical design of experiments (DoE) methods for optimizing materials and chemical processes. It will be demonstrated how BoFire can be applied to experimental setups commonly encountered in the chemical industry: First, it will be shown how incorporating ingredient-specific chemical descriptors through mixing rules in the surrogate model can improve the sample-efficiency of formulation optimization. Second, our recent efforts in optimizing gradient programs in liquid chromatography (LC) with respect to both separation quality and measurement run time will be highlighted. In LC, gradient programs are typically expressed as piecewise-linear functions defined by nodes specifying the organic solvent fraction at specific time points, as required by most chromatographic control software. This representation introduces a many-to-one mapping, since different node sequences may encode the same gradient profile. To resolve this, we use Wasserstein distance-based kernels to compare gradient shapes rather than node sequences, ensuring that equivalent gradients are recognized as identical during optimization.

Abstract: Large language models are increasingly used not just to write code, but to search over it — iteratively generating, evaluating, and refining executable programs as a form of optimization. This talk frames automated algorithm design and AutoML as instances of the same underlying problem: optimization in code space, where the search process explores a space of programs rather than a fixed set of hyperparameters. I will introduce LLaMEA, an evolutionary framework that uses LLMs to generate and refine metaheuristic optimization algorithms, and show how the same ideas extend naturally to AutoML pipeline design. I will then present BLADE, a benchmarking framework for LLM-driven algorithm design, and its recent extension to OpenML tasks, including a leakage-safe dataset summarization protocol that prevents LLMs from exploiting memorized benchmark identities. The talk closes with preliminary results comparing LLaMEA and related search methods (EoH, ReEvo, MCTS-AHD) against strong tabular baselines (AutoGluon, TabPFN) on OpenML classification tasks, highlighting both the promise and the current robustness challenges of code-space optimization for AutoML.

Abstract: Finding the best architecture, training data and hyperparameters for foundation modes is complicated by the large amount of compute required to train these models. So how do model developers make these decisions in practice? In this tutorial, I will provide an overview of the strageties that developers and researchers employ to make judgements about behavior at frotier scale. This includes proxy models and scaling laws, observational analysis, and single-run designs that leverage variation within individual training runs. We will discuss what these strategies are and how they work, and also how they can fail. In addition to discussions about model development, this tutorial will also cover strategies that participitiants can employ in their own research. If time permits, we will also discuss hyperparameter transfer, coordinate checks, and the infinite-width theory of scaling up models. By the end of the tutorial, participants will be able to reason about the benefits and limitations of different research strategies for foundation models.

Abstract: Large language models are increasingly being used not only to generate text or code, but as agents that interact with tools, retrieve information, execute actions, and adapt their behavior in response to environmental feedback. This tutorial introduces the foundations of LLM agents, including tools, planning, state, and memory.

We will explore these components through concrete examples, with a particular focus on web search agents. How do agents decide how to decompose a task, which actions to take, how to use intermediate results, when to revise their approach, and when to stop? These examples illustrate a central challenge: agent performance depends not only on the underlying model, but also on the process through which it acts.

Building on this foundation, we will move from individual agents to orchestrated agent systems. We will introduce core design patterns, including structured workflows and multi-agent coordination. We will discuss when these patterns are useful, how information and responsibilities can be distributed across components, and why adding more agents does not necessarily lead to better outcomes.

By the end of the tutorial, participants will have a conceptual foundation for understanding LLM agents, multi-agent systems, and for reasoning about the efficiency and reliability of agentic systems.

Abstract: Before training a large language model, we need to ensure that our training procedure (e.g. hyperparameter setting, dataset mixtures, model architecture, etc.) works at the target scale. However, since we can only train our model once at full scale, we need to test it on smaller scales first. Scaling laws offer a principled way to predict how a training procedure will behave at scale. By modeling loss as a function of parameters, data, and compute, we can extrapolate from many cheap small-scale runs to the single large model we actually intend to train.

This tutorial introduces neural scaling laws from the ground up, starting with the foundational results of Kaplan et al. and the compute-optimal Chinchilla frontier, the different methodologies used to estimate them, and why early studies disagreed on the optimal trade-off between model size and tokens. I will then turn to scaling laws in practice, using OpenEuroLLM, which is a fully open effort to pre-train multilingual LLMs on European supercomputers, as a running case study. I close with an outlook on automating this process and its connections to AutoML.

Abstract: Hyperparameter optimization (HPO) is a cornerstone of modern machine learning, yet most AutoML systems operate as black boxes: they return a well-performing configuration, but offer little insight into why it performs well, which hyperparameters actually mattered, or whether tuning was worth the effort in the first place. This opacity undermines trust and makes it hard to transfer insights across datasets, tasks, and optimizers.

In this talk, I will give an overview of the landscape of methods for explaining HPO, from ablation analyses and sensitivity studies to notions of hyperparameter importance and tunability, and show how these seemingly different questions can be unified under a single game-theoretic lens. Modeling hyperparameters as players in a cooperative game allows us to use Shapley values and Shapley interactions to additively decompose a performance measure across hyperparameters, yielding both local explanations of individual configurations and global insights into learning algorithms and optimizers. Building on this foundation, we introduce HyperSHAP, a post-hoc explanation framework that instantiates a family of explanation games covering ablation, sensitivity, tunability, and optimizer behavior.

The session concludes with some hands-on tutorial using the HyperSHAP package, computing and visualizing Shapley-based explanations for HPO problems.

Abstract: In the hands-on part of the NAS tutorial, we aim to showcase the importance of an analysis of the proxies beyond a simple correlation with the CIFAR-10 validation accuracy, by focusing on the ability to also include information about hardware constraints, such as latency or energy. We will furthermore compare different network representations, from zero-cost proxies over topological features to using universal text encodings. The participants will learn how different representations and regression methods impact the performance prediction, especially in the multi-objective case and how these proxies can be used to speed-up their NAS methods.