LA-UR-26-28220 Approved for public release; distribution is unlimited. Title: HexML Internship report Author(s): Roperch, Arzhela Carolina Intended for: Report Issued: 2026-09-15 Los Alamos National Laboratory, an affirmative action/equal opportunity employer, is operated by Triad National Security, LLC for the National Nuclear Security Administration of U.S. Department of Energy under contract 89233218CNA000001. By approving this article, the publisher recognizes that the U.S. Government retains nonexclusive, royalty-free license to publish or reproduce the published form of this contribution, or to allow others to do so, for U.S. Government purposes. Los Alamos National Laboratory requests that the publisher identify this article as work performed under the auspices of the U.S. Department of Energy. Los Alamos National Laboratory strongly supports academic freedom and a researcher's right to publish; as an institution, however, the Laboratory does not endorse the viewpoint of a publication or guarantee its technical correctness. HexML Internship report Arzhela Roperch August 28, 2026 1 Introduction The goal of my internship was to continue exploring the use of reinforcement learning for geometry decomposition. This report will therefore describe the investigations I conducted and the challenges I encountered. 2 TensorBoard and Optuna Integration In reinforcement learning, several libraries are commonly used to standardize and simplify the devel- opment of learning pipelines. Among the most widely used are: • Gymnasium: a framework for building and interacting with reinforcement learning environ- ments; • Stable-Baselines3: a library providing robust and widely used reinforcement learning algo- rithms; • RL Baselines3 Zoo: a collection of scripts, hyperparameter configurations, and utilities for training, evaluating, and comparing agents based on Stable-Baselines3. These libraries are also compatible with more general machine learning tools that provide additional functionalities, such as: • TensorBoard: a visualization tool for monitoring learning curves and experiment metrics; • Optuna: a framework for automated hyperparameter optimization. I found really usefull to use them as it standardize a bit the code and provides many tools for the training. 2.1 Tensorboard According to the TensorBoard documentation, Tensorboard provides the visualization and tooling for machine learning experimentation : • tracking and visualizing metrics such as loss and accuracy • visualizing the model graph (ops and layers) • viewing histograms of weights, biases, or other tensors as they change over time • projecting embeddings to a lower dimensional space • displaying images, text, and audio data • profiling TensorFlow programs • and much more 1 More information is available at: https://www.tensorflow.org/tensorboard To integrate TensorBoard into HexML, I implemented a dedicated writer in: relearn/plot utils/tb writer base.py This writer allows specific data to be logged during training and visualized in real time. All TensorBoard logs are stored in the tb logs directory. The training curves can then be visualize in an interface (see fig. 1) Figure 1: Tensorboard visualization interface 2.1.1 Usage I chose to use a custom TensorBoard writer. The TbWriter class, defined in relearn/plot utils/tb writer base.py, initializes a SummaryWriter from the PyTorch library. To use it, you only need to provide the log di- rectory and define the logging functions corresponding to the values you want to monitor. • Using the existing metrics: – Configure the TensorBoard output directory in the script file: OUT_TB_DIR="./tb_logs/Tb_${OUT_DIR_NAME}" Each run will use the same name as the output directory, for example: OUT_DIR="./out_cmapnet-polyg-a4-TEST/" – Once training has started, the logs will be written to the specified path. – You can monitor the training in real time using: module load miniconda3 source activate /usr/projects/hexml/pyoccenv tensorboard --logdir=’log_dir_path’ Then you just need to open the link in your browser. • Adding new metrics: – Modify the TensorBoard writer class to add the scalar values you want to log. – Call the corresponding TensorBoard logging functions from the training code where these scalar values are available. 2 2.2 Optuna Optuna is a hyperparameter optimization framework for machine learning. It is based on two main concepts: • Study: an optimization process based on an objective function; • Trial: a single execution of the objective function with a given set of hyperparameters. The goal is to run multiple trials with different hyperparameter configurations in order to analyze their impact on the objective function. Each study consists of several trials. In general, increasing the number of trials improves the chances of finding a better-performing configuration. Figure 2: General workflow of an Optuna study. A key aspect of Optuna optimization is the choice of the objective function. This function is used as a reference to evaluate each set of hyperparameters and, when necessary, to prune unpromising trials. In the reinforcement learning setting, several objective functions can be considered. For example, one could optimize the cumulative reward or a domain-specific quality metric. In this work, I chose to optimize the topological quality. The main optimization parameters are: • N TRIALS: number of different hyperparameter configurations to sample; • N STARTUP TRIALS: number of trials completed before starting pruning; • MAX EPISODES / MAX STEPS: number of episodes or steps run for each trial; • N EVALUATIONS: number of evaluation points that can be used for pruning decisions; • Pruner: Median Pruner (default); • Sampler: TPESampler (default). When running Optuna studies, choosing an appropriate pruning strategy is crucial. Some training runs may improve slowly at the beginning but eventually reach a better final score. Therefore, pruning trials too early may eliminate promising configurations. Useful references. Optuna for reinforcement learning • Automatic hyperparameter tuning - RL Tutorial ICRA 2022 : https://araffin.github.io/ slides/icra22-hyperparam-opt/#/7/2/1 • Optuna: A Next-generation Hyperparameter Optimization Framework, KDD, 2019. • Github optuna examples - rl - sb3-simple: https://github.com/optuna/optuna-examples/ blob/main/rl/sb3_simple.py 3 2.3 Usage To use Optuna, configure the script located in relearn/experiments/script optimization and run it. This script calls relearn/optimize model.py and creates an output directory containing the information for all trials, as well as an optuna study.db database file. The optimization process can then be visualized with the following command: optuna-dashboard sqlite://experiments/’optuna_dir_path’/optuna_study.db 2.4 Optuna Parallelization To have a good analysis, an Optuna study should typically contain at least 50 to 100 trials. By using parallelization, the total study time could potentially be reduced from several days to only a few hours. Reminder. The Darwin cluster provides three Quality of Service, QoS, options 1 : • Debug QoS: maximum runtime of 4 hours and maximum node count of 32; • Normal QoS: maximum runtime of 10 hours and maximum node count of 16; • Long QoS: maximum runtime of 48 hours and maximum node count of 8. The question is then: which type of parallelization can be used? The Optuna documentation describes several possible approaches. Figure 3: Possible Optuna parallelization strategies. Source: https://optuna.readthedocs.io/en/ stable/tutorial/10_key_features/004_distributed.html Several parallelization strategies were considered: • Multi-thread: this option is not ideal. Multi-thread optimization has traditionally been ineffi- cient in Python due to the Global Interpreter Lock (GIL) – Or need Python >= 3.14 • Single-node multi-process: this approach caused issues in practice. In particular, multiple processes attempted to access the GPU simultaneously, which led to CUDA initialization errors. A typical error message was : CUDA error: initialization error CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging, consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. 1 https://darwin.lanl.gov/doku.php?id=faq:qos 4 • Multi-node multi-process: for multi-node optimization, Optuna recommends using RDBStorage. This requires a database such as MySQL or PostgreSQL. At the time of the experiments, I didn’t found it, but maybe darwin has one. For these reasons, Optuna trials are currently run sequentially. A possible improvement would be to use a PostgreSQL database on the Darwin cluster. This could enable Optuna’s RDBStorage backend and make multi-process or multi-node studies easier to manage. 3 Reward function investigation The reward function is a key element in modeling a reinforcement learning problem (RL). Indeed, it guides the agent throughout the learning process by indicating whether the actions taken are appro- priate or not. It must therefore be carefully chosen to provide the agent with learning signals that are consistent with the desired objective. In our geometric decomposition problem, the goal is to decompose a given geometric shape into quadrangular shapes. Ideally, the resulting decomposition should consist solely of quadrilaterals, achieved in the fewest possible steps. Furthermore, these quadrilaterals should have good geomet- ric properties: they should be as orthogonal as possible and have sizes that are as regular as possible. The central question, then, is this: How can we construct a reward function that takes these various criteria into account? To answer this question, we experimented with several formulations of the reward function. 3.1 LDQNTri The actual reward function of HexML is the following : qP π 2 qj (θqj − 2 ) R = wq Nq − wt Nt − wns δ1N − w0 2π Here, the goal is to reward the creation of quadrilaterals. We therefore introduce a positive term proportional to the number of quadrilaterals formed, denoted wq Nq . On the other hand, we definitely do not want to create triangles; we will therefore penalize their creation with a negative term −wt Nt . Next, as mentioned earlier, we want the agent to perform the decomposition in a minimum number of actions, while effectively producing a new shape at each step. Thus, when an action does not generate any new shape, a penalty is applied, denoted by −wns δ1N . Finally, the generated shapes should be as orthogonal as possible. To achieve this, we add a penalty term thatr measures the deviation of the quadrilaterals’ angles from a right angle. This term is written π 2 qj (θqj − 2 ) P as: −wo 2π . Thus, the reward function combines several objectives: promoting the creation of quadrilaterals, avoiding the formation of triangles, limiting unnecessary actions, and encouraging the production of shapes close to orthogonal quadrilaterals. The best trial in an optuna study achieves 0.74 of topological quality. 3.2 LDQNneg In reinforcement learning, some approaches involve maintaining a purely negative reward throughout the episode and assigning a positive reward only at the end of the episode, when the agent reaches a winning state. The idea is to penalize each action based on its quality. The instantaneous reward can then be defined as follows: qP π 2 Nq Nt qj (θqj − 2 ) R = wq ( − 1) − wt − wns δ1N − w0 N N 2π In this formulation, each term tends toward 0 when an action produces a high-quality quadrilateral shape with angles close to orthogonality. Conversely, the lower the quality of the generated shapes, the greater the associated penalty. 5 Thus, the agent is encouraged to avoid poor actions at each step, without receiving any intermediate positive reward. If the agent succeeds in completely decomposing the geometry into quadrilaterals with good orthogonality, a final positive reward +rf is then awarded. This positive reward will be reflected in the preceding actions when the weights are updated. Actions leading to correct final decompositions will then be more heavily rewarded. Note: I’m wondering wether I did a mistake in this reward function. I’m scared I might have rewarded in the cases the episode ends with an invalid decomposition (with triangles), which should be penalize instead. The best trial in an optuna study achieves 0.73 of topological quality. 3.3 LDQNTopoImpr If our evaluation criterion is the topological quality of the resulting decomposition, why not incorporate it directly into our rewards? We can thus consider a reward that incentivizes the agent when the topological quality improves, and penalizes it when it deteriorates. The topological quality is defined by : Nq + Ngq − Nt − 0.25Ng Q= Ns The reward at step t + 1 can then be defined as: Rt+1 = Qt+1 − Qt , where Qt+1 denotes the topological quality obtained after the action is performed, and Qt denotes the topological quality before that action. This formulation has the advantage of directly seeking to improve the topological quality of the final decomposition. However, it does not take into account the geometric quality of the generated shapes, particularly their degree of orthogonality. Limits: The topological quality is not enough to differentiate some decompositions. The same topological improvement can reflect really different situation, a good action can lead to exactly the same improvement as a bad one, as shown in figure 4. Figure 4: Three actions leading to a same topological improvement but really different decomposi- tions. The first quad in the top-right image is technically invalid (one node is not at a corner vertex). However, the concept illustrated remains correct, by moving this node slightly upward would produce a valid quad with same behavior. I didnt had time to change the figure. 6 3.4 LDQNTopo Another possibility would be to define the reward solely based on the current topological quality of the decomposition. However, this approach does not seem particularly relevant to me, since the same action can yield very different rewards depending on the configuration in which it is performed. For example, if the current decomposition has low topological quality, with q = 0.1, and the agent creates a new quadrilateral, the reward obtained may remain relatively low, for example r = 0.25. Conversely, if the agent is nearing the end of an episode, with a decomposition that is already of good quality—for example, q = 0.7—then creating a single additional quadrilateral can lead to a much higher reward, for example, r = 0.85. Thus, two actions that are similar and equally relevant may be associated with very different rewards. The reward then depends more on the overall state of the decomposition than on the intrinsic quality of the action performed, which, in my opinion, can make the learning signal less consistent. 3.5 LDQNorthoas One might also ask why we consider only the orthogonality of the quadrilaterals, and not that of all the shapes present in the decomposition. One initial idea is to add an orthogonality penalty for both the created quadrilaterals and the remaining general shapes: qP qP π 2 π 2 qj (θqj − 2 ) gj (θgj − 2 ) R = wq Nq − wt Nt − wns δ1N − w0 ( + ) 2π 2π I started with this formulation, but a problem quickly arises: general shapes often have many more an- gles than quadrilaterals. Their orthogonality penalty can therefore become much larger and dominate that associated with the created quadrilaterals. In this case, the reward signal risks being primarily influenced by the geometric quality of the remaining shapes, rather than by that of the quadrilaterals produced by the agent. To limit this effect, a second version involves normalizing the penalty for general shapes by their number of vertices Nv : qP qP π 2 π 2 qj (θ qj − 2 ) gj (θgj − 2 ) R = wq Nq − wt Nt − wns δ1N − w0 ( + ) 2π 2π ∗ Nv where Nv denotes the number of vertices of the general shape under consideration. However, I am not sure that this addition provides any real benefit. Indeed, does the agent actually have direct control over the orthogonality of the remaining general shapes? Its primary objective is to create quadrilaterals. It therefore seems more natural to encourage it to produce quadrilaterals that are as orthogonal as possible. As the agent generates high-quality quadrilaterals, the final decomposition should gradually converge toward a structure composed of quadrilaterals that are themselves well- orthogonal. Consequently, adding a penalty for the orthogonality of other shapes risks obscuring the information related to the quality of the created quadrilaterals. The learning signal could then be dominated by angles belonging to shapes over which the agent has only indirect control—or even very limited control in the short term. 3.6 Comparative results We compared the training performance obtained with the different reward functions. The only function not included in this comparison is LDNNeg, since it was evaluated separately during the Optuna studies. Figure 5 shows the evolution of both the topological quality of the decompositions and the rewards throughout training. 7 Figure 5: Comparison of the different reward functions. Based on these results, using an alternative reward function does not appear to provide any clear advantage. 4 Part-GeoNet Agent Part-GeoNet agent is a reinforcement learning (RL) agent based on the Proximal Policy Optimization (PPO) algorithm [SWD+ 17]. It uses a GNN to construct a representation of the geometries to be segmented. To segment the geometries, the agent needs information that goes beyond the local neighborhood of a vertex. This is because each segmentation action performed by the agent will have a global impact on the geometry. The agent must therefore be able to take into account both local information and more distant geometric information. How can this distant information be easily integrated? GNNs are characterized by their ability to propagate information between neighboring nodes at each layer. The number of layers in the GNN thus determines the distance over which information propagates. The more layers added, the farther the information will propagate. However, this propagation occurs through the topological adjacencies between vertices. Regions that are geometrically close but topologically distant will not be reachable. Increasing the number of layers in the GNN is not necessarily a solution, however. For large geometries, capturing all the global information would require a very deep network, which would make the model difficult to scale. Furthermore, too many layers can lead to a phenomenon known as oversmoothing, where the representations of the nodes gradually become too similar and lose their discriminative power. However, the structure of GNNs makes it possible to consider more sophisticated architectures. For example, the use of multi-graph GNNs would allow for more direct access to information that is topologically distant but necessary. That is why we have designed a learning agent that incorporates a GNN to represent the charac- teristics of the geometries to be cut. 4.1 General agent framework Our learning model, Part-GeoNet, is based on the Proximal Policy Optimization (PPO) algorithm. It operates in four steps: 1. Encode the vertex features into a latent space using a GNN 2. Predict the value of the current state (critic) 3. Select the vertex from which to perform a cut (actor) 8 4. Predict the type of cut to perform (actor) Through Part-GeoNet, our agent will then learn both a latent representation of the vertex features and a global representation of the decomposition. The local representation of the vertices is used by the actor to select a vertex and perform a cut. The global representation, on the other hand, is used by the critic to estimate the expected rewards, as illustrated in Figure 6. Figure 6: Based on the graph representing the geometry, we will use a GNN to generate latent representations of the geometry. These representations will then be used by the actor to predict the cut and by the critic to estimate the expected rewards based on a state. Below, we’ll go into detail about each of these steps. 4.1.1 GNN Embedding: We are operating within a partially observable environment; the agent has access only to certain information about the state in which it finds itself. The actor can do with a local observation of the shape it need to decompose. In contrast, the critic needs more comprehensive information about the overall decomposition to correctly predict the expected rewards. To address this constraint, the GNN will encode geometric information in two forms. First, for each vertex, it will generate a latent representation through local aggregations of neighboring vertices. Second, to obtain a more global view, it will combine information from the complete decomposition with an aggregation of information across all vertices of the current shape being processed. This construction enables the implementation of an asymmetric actor-critic architecture, in which the actor and the critic receive different observations, as illustrated in Figure 7. • The critic will use global information about the decomposition state to estimate the rewards it can obtain from that state. • The actor, on the other hand, will rely on a more local observation of the current shape to be decomposed. 9 Figure 7: Based on the graph representing the geometry, the GNN generates two types of representa- tions. The first, based on vertices, is constructed through an intuitive aggregation of the characteristics of neighboring vertices, followed by a second layer of average aggregation to obtain the latent repre- sentations of each vertex in the geometry. The second is a global representation of the geometry: it is obtained by aggregating information from all vertices, then combined with global information about the decomposition. The technical details of the aggregations used evolved throughout the internship. The different implementations are presented in sections 4.2 and 4.3. 4.1.2 Critic estimation: Based on the global representation of the geometry, the critic estimates the value of the current state, that is, the expected rewards from that state. The estimation is performed by a two-layer MLP with 128 neurons per layer. 4.1.3 Node selection: Based on the vertices latent representations produced by the GNN, the agent predicts a value for each vertex. These values are transformed by the softmax function into a probability distribution. From this distribution, we sample the vertex to be selected. The prediction of these values is also performed by a two-layer MLP with 128 neurons. 4.1.4 Cutting choice: Once the vertex is selected, its latent representation is reused to predict the type of cut to be applied to the previously selected vertex. The choice of cut is also performed by a two-layer MLP with 128 neurons. 4.2 Brep Graph Part-GeoNet The first GNN that comes to mind would be the B-rep graph of the geometry, that is, the graph associated with the edges and vertices on the boundaries of the geometry (see Figure 8). 10 Figure 8: Brep graph from a geometry This graph has the advantage of providing a simple and direct representation of the geometry. However, its limitations remain relatively similar to those of a manually constructed local observation. Indeed, with a reasonable number of layers, i.e., approximately 3, the GNN cannot propagate informa- tion beyond a distance of 3 neighboring vertices. Information located at the other end of the geometry will therefore still not be captured in a vertex’s local representation. Despite this limitation, this approach provides us with an initial point of comparison. Before using more complex graphs, we wanted to ensure that this approach yields results similar to those of previous observations. The specificity of this Brep Graph GNN is that the first layer is an intuitive aggregation to focus on the main features and avoid the oversmoothing issue of GNNs. 4.2.1 GNN fisrt layer intuitive aggregation As said befoer, the subtelty of PartGeoNet is that the first layer of the GNN is an intuitive aggregation. Each vertex i has the following characteristics: xi = [vtype , α, vcx , vcy , v1x , v1y , v2x , v2y , A, ar , nside , nend , ncorner , nreverse ] The aggregation function is decomposed into three components:   Agg(Xi ) = CONCAT xi , fneigh (Xi ), fshape (Xi ) (1) K   (k) M (k) (k) (k) (j) (j) Agg(Xi ) = xi ∥ [vtype , α , vx , vy ] ∥ v̄type , min α , max α (2) |{z} j∈X j∈X k=1 node features| {z } | {z } full shape (partially global) 2 neighbors vertices (local) (1) (2) Agg(Xi ) = xi ∥ vtype , α(1) , vx(1) , vy(1) ∥ vtype , α(2) , vx(2) , vy(2) ∥ v̄type , αmin , αmax (3) Notation. • xi ∈ Rd : intrinsic node features, • Xi : the set of all neighbors of node i, • X : the set of all nodes in the current shape, • ∥: vector concatenation, • L : ordered concatenation over the 2 neighbors. Local features. For each of the 2 neighbors, we encode four edge attributes: (k) • vtype ∈ {−1, 0, 1, 2}: indicator for vertex k type, • α(k) ∈ [0, 2π): relative angle at vertex k, (k) (k) • (vx , vy ) ∈ R2 : normalized vector from vertex k to its neighbor. 11 So, we have K (k) M fneigh (Xi ) = [vtype , α(k) , vx(k) , vy(k) ] k=1 Global shape features. To obtain more global information of the current shape, we compute three attributes: 1 X (j) v̄type = vtype (mean of vertices type) |Xi | j∈Xi αmin = min α(j) , αmax = max α(j) (angular span) j∈X j∈X So, we have fshape (Xi ) = [v̄type , αmin , αmax ] 4.2.2 GNN second and after layers For the next layers, I used mean aggregation layer concatenated with the nodes features 1 X Agg(Xi ) = xi ∥ xj (4) |{z} |Xi | node features xj ∈X i | {z } mean agg 4.2.3 Advantage and limits The advantage of this aggregation is that the first layer focus on the main features of the node and we can perform intuitive combinations that fits our problem and generate fixed size vector. So our first linear layer of the GNN have a concentrated main feature vector with as many concatenated features as we can. As we know every vertex has only two neighbors, it is easy to concatenate. With this architecture, a one-layer GNN is functionally equivalent to manually extracting features from the local neighborhood. 4.3 Brep and visibility Graph Part-GeoNet To improve the observation provided to our agent, we want to enhance the geometry representation by adding informations about visible vertices. A vertex is said to be visible from another vertex if the segment connecting these two vertices encounters no obstacles. We can then construct a visibility graph connecting the vertices that are visible to one another, as illustrated in Figure 9) Figure 9: The visibility graph connecting the visible vertices one to one another. Using the visibility graph allows the agent to obtain additional information about areas of the geometry that are topologically farther away. To exploit both topologically close and distant informa- tion, we construct a GNN multigraph in which the two graphs,BREP and visibility, share the same vertices but have different adjacency relationships. In practice, to construct this multigraph, we can use the HeteroData class from PyTorch Geometric. This class allows us to have different types of nodes and different types of edges within the same 12 graph. In our case, we use a single node type: vertex, and two edge types: brep and visibility, corresponding respectively to the edges of the BREP graph and the visibility graph. To take full advantage of this multi-graph, we have several options for aggregating each layer : • An aggregation of all neighboring vertices without distinction of the type of edge. • Two separate aggregation for each edge type, then concatenate the results • Two nested aggregations. First we aggregate the information from the visible vertices and then concatenate the result with the characteristics of the neighboring vertices of brep graph. The concatenated result is then passed to a linear layer. We chose to use the last method: the information from the B-rep graph and the visibility graph is aggregated separately, then combined so that the linear layer can produce the final representation of each vertex. 4.3.1 GNN Aggregation Aggregation on the multigraph is divided into two parts: First, we will aggregate the features of the vertices visible via “visibility” edges using an attention layer. Next, we will concatenate the result with the features of the neighbors via BREP edges to perform the final aggregation. Adding the attention layer allows us to better distinguish between visible vertices. In fact, a simple mean aggregation layer can cause a phenomenon of feature smoothing, known as over-smoothing. For large geometries, vertices may have many visible neighbors, and the resulting mean may be nearly identical. The information provided by the visible vertices would then be of limited discriminative power. The attention layer, on the other hand, learns to assign different weights to each neighbor. This assigns varying levels of importance to the vertices and should generate better representations. Continuing to denote the characteristics xi of vertex i, and Xi the set of characteritics of the graph’s vertices, we will add a fourth component to the aggregation function :   Agg(Xi ) = CONCAT xi , fneigh (Xi ), fshape (Xi ), fvisi (Xi ) (5) This new component fvisi (Xi ) corresponds to the aggregation of visible vertices by an attention layer. The attention layer used in our implementation is PyTorch’s Geometric GATConv, whose method is illustrated in Figure 10. Figure 10: Graph attention layer process extracted from the original paper [VCC+ 18]. 13 With this structure, the agent has access not only to information about its topological neighbors but also to more global information from the geometry’s visible vertices. 4.4 Proximal Policy Algorithm Proximal Policy Optimization (PPO) is a Policy Gradient algorithm developed by OpenAI [SWD+ 17]. It is derived from the TRPO (Trust Region Policy Optimization) algorithm [SLM+ 17], which delivers good performance but requires a more complex implementation. PPO achieves the performance and reliability of TRPO while using a simpler first-order optimization. TRPO relies on the probability ratio between the current and old policies, πθπθ (a(at ,st ,st )t ) , which mea- old sures the change in the policy. The associated objective function is written as:   πθ (at , st ) LCP I (θ) = Eπ Ât πθold (at , st ) where θ denotes the parameters of the policy network, and Ât is the estimated advantage function. PPO differs from TRPO in that it clips the ratio by a “factor” ϵ. This ratio helps prevent updates from becoming too large. This function is then called the “Clipped Surrogate Objective”:   πθ (at , st ) πθ (at , st ) LCLIP (θ) = Eπ min( Ât ), clip( , 1 − ϵ, 1 + ϵ)Ât (6) πθold (at , st ) πθold (at , st ) Where ϵ is the parameter used to adjust the clipping of the function. In fact, the second term of this function, clip(ln π(at , st ), 1 − ϵ, 1 + ϵ)Ât prevents the parameters from being updated too drastically, thereby limiting the risk of changing them too much all at once and causing the agent to head in the wrong direction. This objective is called pessimistic because it preserves the lower bound between the unclipped objective and the clipped objective. 4.4.1 SB3 PPO Implementation The original PPO code has been made available in the GitHub repository openai/baselines. However, there are many implementations of the algorithm with varying performance. The SB3 implementation has the advantage of being easy to use, well-documented, and detailed. I therefore drew inspiration from it to create my own version. Shengy et al. [HDR+ 22] compared OpenAI’s PPO implementation in detail with other implementa- tions, such as SB3. We will follow their main points of comparison. 1. Weight Initialization The biases are initialized √ to 0. The weights are initialized using the or- thogonal scaling method with a scale of 2 for the hidden layers. The output layer of the actor network is scaled to 0.01, and the critic’s to 1. This orthogonal initialization generally yields better performance than Xavier initialization [EIS+ 20]. How did I actually do it? 2. Advantage estimation We use the Generalized Advantage Estimator (GAE) [SML+ 18]. et al. [ARS+ 21] found that using GAE performs better than T D(λ) 5 New environment To use Part Geo Net, we decided to experiment with a new environment. The previous environment had several aspects that I couldn’t fully explain. In particular, three elements seem essential to me for building a coherent learning environment: • A good reward function. This guides the agent in its learning; the signals it sends must be clear and consistent with the intended goal. • A sufficiently rich set of actions. The set of actions must contain a variety of actions sufficient to solve the problem. • Sufficient observation. The observation must be detailed enough to allow the selection of appro- priate actions. 14 From these three elements follows a fundamental principle in RL: • The same observation-action sequence must always produce the same reward. And this reward must be consistent with the objective. The agent’s observation must therefore be detailed enough to distinguish between different scenar- ios. The problem we are trying to solve is all the more complicated because it is multi-objective. In fact, the agent must first successfully select the most relevant vertex for a cut and then choose the appropriate cut. But if the initially selected vertex is incorrect, the cut cannot be correct. With the current observation and set of actions, I found it difficult to choose a vertex and even more difficult to choose the cut. An example is shown in Figure 11 Figure 11: How do you determine the best cut in this configuration? The red action pointing downward seems more relevant, but what distinguishes it from the red action at the top? Similarly, which vertex was the best choice to begin with? With the current local observation, nothing distinguishes the three vertices with obtuse angles. These questions motivated the investigation of a new environment. The criteria for this new environment: • Use a set of actions that seems intuitive • Define an observation that allows us to differentiate between good and bad actions, in particular by: – expanding the observation to better capture distant features of the geometry in order to obtain sufficiently distinguishable observations • Studying the geometry dataset and considering cleaning it up 5.1 Action Set The newly selected action set includes two types of cuts: • Perpendicular cut: The cut starts at the selected vertex and follows the line perpendicular to one of the adjacent edges. • Cut at third nodes: The cut starts at the selected vertex and connects to the third nearest vertex on either side. 15 Figure 12: New action set. It consists of the old perpendicular cuts in red and the new third nodes in orange With this action set, it seemed easier to me to define local heuristics for selecting actions. 5.2 Observation This new set of actions required us to expand the agent’s observation. We felt it was appropriate to add information regarding the two third neighboring vertices. So we enriched our observation with: • α1 and α2 , the angles at the selected node in both of the newly generated faces, • v⃗3p and v⃗3n , the vectors pointing to the previous and next third nodes, respectively, • β1 and β2 , the interior angles of the third nodes, • whether the two third nodes are visible from the vertex in question. See Figure 13 for further clarification. Figure 13: Illustration of the new information added to the features of a vertex. The final features associated with each vertex are therefore as follows: xi = [vtype , α, vcx , vcy , v1x , v1y , v2x , v2y , A, ar , nside , nend , ncorner , nreverse , x y visi α1 , v3n , v3n , β1 , v3n , x y visi α2 , v3p , v3p , β2 , v3p ] These features are then processed by the GNN to produce a representation that can be used by the agent. 16 5.3 Reward function Adding cuts toward third-neighbor nodes revealed an inconsistency with the previous reward function. Indeed, a third-node cut always creates a quadrilateral, and is therefore systematically rewarded. As a result, the agent tends to learn a strategy that selects only third-node cuts, producing final shapes that necessarily lead to triangles. This behavior is illustrated in Figure 14. To avoid this phenomeon, we tried to focus more on the orthogonality of the shapes. The new reward function introduces an orthogonality score of the entire decomposition. This score is computed as the sum, over all interior angles, of their deviation from 90◦ : X π sortho = (|θi − |) 2 i∈N The agent is then rewarded according to the improvement in this orthogonality score between two consecutive steps: sortho (t) − sortho (t + 1) his quantity is positive when the decomposition becomes more orthogonal, and negative when the orthogonality deteriorates. However, I found out that using only orthogonality criterion as reward led to many triangles shapes. To address this issue, I added the initial penalties for triangle creation and invalid cuts, while keeping a small positive reward for the creation of quadrilaterals : R(t + 1) = ws (sortho (t) − sortho (t + 1)) + wq Nq − wt Nt − wns δ1N The new reward function seems to provide more consistent rewards than the old one. Figure 14: Reward inconsistency. I used the following weight. For the old reward, wq = 50, wt = 50, wns = 20. For the new wq = 5, wt = 20, wns = 20 and ws = 10. 6 Results Note: All curves in this report can be viewed in greater detail in the TensorBoard logs located in /usr/projects/hexml/arzhela results. The folders specific to each section will be indicated in the figures. 17 6.1 Datasets The dataset used for all these experiments is “planar poly general,” excluding the file poly76.step, since it is a simple quad and caused an error at the beginning of the episode because I had not anticipated this scenario. 6.2 First Results: PartGeoNet I first implemented a very simple version of Cmapnet—the agent I had created for geometric smooth- ing—using a 3-layer GNN and a simplified version of PPO. I then gradually improved the PPO imple- mentation. The initial algorithm was limited to implementing the PPO objective function presented in equation 6, with a loss function shared between the actor and the critic. 6.2.1 PPOImpr - Add entropy and value clipping First, I added an entropy term as described in the paper [SWD+ 17] to obtain a loss function of the following form: LCLIP +V F +S (θ) = Êt LCLIP (θ) − c1 LVt F (θ) + c2 S[πθ ](st ) ,   t where c1 , c2 are coefficients, S denotes an entropy bonus, and LVt F is a squared-error loss (Vθ (st ) − Vttarg )2 . I then added gradient clipping before the updates, as well as value function clipping. PPO clips the value function in the same way as the PPO’s clipped surrogate objective. (No study has proven that this improves performance. M. Andrychowicz et al. [ARS+ 21] even find that it degrades performance.) 6.2.2 PPO Asymmetric Observation After thinking on our agent’s ability to predict the expected rewards of an episode, it seemed to us impossible to predict the expected rewards by limiting ourselves to the initial local observation. Indeed, two similar local states can appear at the beginning as well as at the end of an episode. However, the cumulative rewards necessarily depend on the progression of the episode. This is why we wanted to provide the critic with a more global observation. The critic’s observation will then have access to the following global information, in addition to the local features of the vertices of the current shape:   xglobal = nb parts in queue, nb verts in model, nb quads, nb tris, nb genquads, nb genshapes, topo quality The actor, on the other hand, does not need information about the global decomposition. It can limit itself to the shape selected for cutting. That’s why we tried an asymmetric architecture. 6.2.3 Networks losses. We used two separate networks for the actor and critic, but the loss function were initially combined. By default in SB3, the global loss is the weighted sum of the actor loss, critic loss and entropy bonus. The respective weights are set by default to : 1.0 for the actor loss, 0.5 for the critic and 0.005 for entropy. The problem we encountered with this combined loss was that the critic loss was disproportionately large compared to the actor loss. Indeed, the rewards ranges from -1000 to +1000 over an episode. The critic’s predictions, which initially lie in the range [-1, +1], result in enormous squared errors when compared to these large-scale returns. When comparing the orders of magnitude of the different loss components (see fig. 15), the imbalance becomes evident. 18 Figure 15: While the actor loss is close to 0, the critic loss reaches 1e+5. I tried various approaches of adressing this problem. 1. Normalize the returns. Claude code recommended normalizing the returns to bring them to a comparable scale. While it improved the learning, it is conceptually wrong. For the actor loss, we compare the actual reward obtained to the critic value estimation. If the critic learns on normalized values, the predictions will not reflects anymore the reward scale, corrupting the advantage calculation. 2. Separate actor and critic training. I seperate the two loss functions and their optimization. The actor and the critic are then trained independently, with updates based on their own loss signal. 3. Rewards scaling. To help the critic learning the value, I rescales the rewards to bring values closer to the range [-1,1]. 6.2.4 Results tb logs polyg ppo improvement Figure 16: Topological quality of the decomposition over the learning episodes. The version of PPO with independent actor and critic and asymmetric observation seems to yield slightly better learning results. So we continued working with the improved PPO, using asymmetric observation and independent training for the actor and critic. 6.3 GNN Layers Investigation Another important aspect while construction the GNN architecture is to select the appropriate number of layers for the GNN. It is typically recommended to have 2-3 layers, as deeper architecture can suffer from over-smoothing. I conducted experiments on both datasets, planar rect and planar poly general. However, sadly theses experiments are not reliable as at this time the seeds were not fixed during train- ing. 19 tb logs GNN layers investigation Figure 17: Topological quality of the decomposition over the learning episodes with 1, 2, 3, 4 and 5 layers. I conclude I would use 2 layers in the following learning 6.4 Multigraph PartGeoNet on previous environment After implementing PartGeoNet Multigraph, which combines the BREP graph and the visibility graph with a neighbor-based attention layer, we compared the results with those of the previous agent, SAC- Splitter. The comparison was conducted in the initial environment using the planar poly general dataset. Unfortunately, all of the initial results contained an error in the attention layers that signifi- cantly degraded the results... Just before the end of my internship, I was able to correct the error and obtain the following results: tb logs cmapnet old env Figure 18: Topological quality of the decomposition over the learning. PartGeoNet seems to learn a bit better. The maximum topological quality reached by PartGeoNet was 0.78 while for SAC it was 0.73. 20 Figure 19: Percentage of each action type chosen by the agent during learning. PartGeoNet seems to choose slightly more different actions, but tends to the same distribution, almost 100% of perpendicular cuts. Figure 20: Type of shapes created by the agent during learning. PartGeoNet seems to create less triangles, while creating less shapes. These results are interesting but I’m wondering what would have happened if we let the agent train longer. Probably it would have just converges to a similar value, but I’m still curious because of the last curves tendance. 6.5 New environment and Multigraph PartGeoNet We then applied PartGeoNet to the new environment. This turned out to be quite complicated and unsatisfactory. In fact, the “third node cut” action consistently generates a new quad whenever it can be performed. This makes it the ideal action from our agent’s perspective. By performing this action, the agent will always obtain a quad—and thus a reward. To avoid this problem, I chose to create a reward function based on the orthogonality of the interior angles. However, this approach also has a problem: the triangles are not penalized enough. Consequently, I believe the rewards are consistent, but the topological quality tends to be too low. 21 tb logs cmapnet old env Figure 21: Topological quality of the decomposition over the learning. PartGeoNet on the new envi- ronment performs worse than SAC Splitter on the old environment. 6.6 Main evaluation results The evaluation of PartGeoNet is stochastic. The agent plays 10 times on a geometry using the final stochastic policy. This yields 10 different results. We finally select the best one from among them. The evaluation of SAC is different: the agent plays once on each geometry using a deterministic policy. NOTE: This may also explain the differences in results; it is very likely that this evaluation method strongly favors PartGeoNet over SAC. Figure 22: Evaluation results on the test dataset for PartGeoNet on the old environment and on the new environment. PartGeoNet on the new environment seems to have 6 shapes definitely better or as good as on the old environment. Old SAC has only one shape performing better than the others. PartGeoNet on the old env is the one perfoming better with 7 best shapes. 22 7 Conclusion After familiarizing with the existing code and integrating TensorBoard and Optuna, we explored various reward functions. Unfortunately, the results were inconclusive, and no reward function seemed to outperform the initial one. We therefore decided to completely overhaul the architecture of our RL agent to use PartGeoNet. PartGeoNet is based on the PPO algorithm and uses a GNN to create a latent representation of the state. This representation is then used to predict the cuts to be made. After several adjustments and improvements to the PartGeoNet structure, the results proved to be slightly better during both training and deployment. However, given the very marginal improvement, it might be worthwhile to make more significant changes to the problem formulation. 8 Perspectives Following this work, we can consider various perspectives: • Incremental learning. Train the model progressively on datasets of increasing difficulty. • Improve evaluation. Add new metrics during evaluation. For example, during stochastic evaluation, it would be interesting to know the mean and standard deviation of the topological quality obtained for each geometry. It might also be interesting to evaluate the meshability of the resulting decomposition. • Meshability as an objective. How could we define a meshability index? Could we use this criterion for learning? • New observations. Use the existing multi-graph GNN structure to test different observations. • Analysis of vertex selection. It would be interesting to add metrics to understand and evaluate the vertex selection made by the agent. • Templates. Use a set of template-based actions to subdivide the geometry. • GNN architecture with link prediction. We could construct the BREP graph of the ge- ometry, then add vertices at singularities and other points of interest, and then use a GNN to predict edges between the vertices, with each new edge corresponding to a cut. References [ARS+ 21] Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? a large-scale study. In International Conference on Learning Representations, 2021. [EIS+ 20] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo, 2020. [HDR+ 22] Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. In ICLR Blog Track, 2022. https://iclr-blog-track.github.io/2022/03/25/ppo-implementation- details/. [SLM+ 17] John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization, 2017. [SML+ 18] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation, 2018. [SWD+ 17] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. 23 [VCC+ 18] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks, 2018. 24