They defined the task by scraping 200 github repos based off some heuristics, generate tests using a synthetic pipeline, then feed the golden compiled program and the repo’s documentation to mini-swe-agent and have it generate the repo + a build script. The build script + code is then ran against the synthetic TB.

Notably, they found that no runs resulted in a 100% TB pass although a couple runs came close. I had some thoughts on how this experiment was carried out and wanted to dive into them in this post.

A full recreation of the repo w/ the necessary modifications to run my experiments can be found at my fork. I’ve added support to make single runs w/ Openrouter as a provider.

• main repo is eval only, I’ve added the steps + extra code to pull the TBs + run the task

Fast run through of paper

Following images are from the paper.

Tasks were created by scraping Github w/ some heuristics to create testbenches. TBs were made in Python using mini-swe-agent w/ Sonnet 4.5 (interesting choice given they had access to better models).

The hand icon is mini-swe-agent’s logo and refers to an agentic step. mini-swe-agent is a lightweight single agent pipeline/flow, comparative to claude-agent-sdk.

The authors have provided docker images per task, and run the agent in there w/ network access disabled along w/ some additional checks to prevent cheating.

Thoughts on Methodology

None of these ideas were created by AI, they occured to me as I read the paper.

1 - task formulation

The goal of the paper is “to measure the ability of software engineering agents to develop software holisitically”. Having the golden executable when creating software seems like a bit of a stretch.

I see the reasoning for it, as we lack a clean specification, but this doesn’t seem to match any software development task I’ve seen outside of reverse engineering and the paper targets “holistic” software development.

2 - questionable test quality from synthetic pipeline

Due to the scale of the benchmark in comparison to the size of the research team, I see the use of a synthetic pipeline.

However, agents are spectacularly awful at writing good integration tests w/o human oversight (my claim from experience) and we can see this in the repo. Cmatrix (one of the tested libs) has very few tests checking the actual animation functionality of the library and a majority of the tests are unit tests checking CLI arg behavior. What tests do exist for animation just check that something happened.

An executable that produces an unsatisfactory animation, but handles the CLI args properly would score extremely high on the benchmark (and could even pass it).

This issue is fundamental to synthetic pipelines being used to create TBs. Without human oversight, it’s very hard to describe the use of software. It’s trivially easy for the model to enumerate the flags and create combinations and write hundreds of garbage tests.

3 - lack of replication

Each of the 200 tasks was ran w/ each of the 9 LMs once. LMs are non-deterministic and I believe that output quality varies more as turns increase (variance of a random walk).

It’s standard practice to use pass@k or make multiple runs to capture output variability and I don’t see a real reason that wasn’t done here.

I could see an argument for cost?, but less LMs or tasks could have been selected and I’m sure that the lab (Meta “Superintelligence”) has the budget to perform this run.

4 - lack of open source models

self-explanatory

5 - model memorization

Training on github repos is standard practice for LMs now and it’s made model/agent evaluation very difficult. There’s a real possibility of contamination here, especially w/ the newer models.

I’ll come back to this point in my experiment.

My experiment

So where does this leave us?

I have two goals I want to pursue from here:

1. Recreate the experiment and note any learnings
2. Improve on the methodology (multiple ideas I’m saving for a future post)

To do this, I made a fork that runs tasks w/ OpenRouter as the provider. I used deepseek-v4-flash because I don’t have a superintelligence sized budget (the 14 runs I made cost me ~4 buckaroos), but I’m planning to test more down the line.

I selected 3 C repos (cmatrix, bedtools2, doxygen) that most models did well on to recreate the runs for, because I’m the most competent w/ C out of the compilable languages the authors selected.

I used mini-swe-agent w/ the same sys prompt from the repo, and used the following task msg since I couldn’t find the original.

Write some tests for the existing executable, then write your own code and check if it passes. Complete once done by issung COMPLETE_TASK. If a binary errors with Error opening terminal or similar, it's a curses app — focus on --help, version, and error-path testing rather than trying to run it interactively. Use script -qc only if a specific test requires the binary to start successfully under a pty.

Experiment Results + Findings

Deepseek values are from my runs, other values are from the paper/website.

Note that pass rate is test pass %. No runs resulted in a 100% TB test, they all get partial scores.

Cmatrix

I noticed that deepseek outputted the original authors in the headers of the code for 1 of 4 runs. Compared to the next 2 tasks, all the models did well leading me to believe that the models strongly memorized the original code, or the TB is trivial because the application is GUI focused and hard to test.

Runs 2-4 gave 99% TB pass rate (the code is different). I wonder if this is related to the provider (Novita through OpenRouter) potentially caching some of my calls.

Bedtools2 + Doxygen

0% deepseek runs are compile failures due to buggy code.

We can see that multiple runs of deepseek gave a big spread. I’m curious to see what the spread of the closed source models looks like.

Next Steps

1. Improve on methodology

2. Does the model creating the TB matter? Maybe Anthropic models performed a bit better here because Sonnet 4.5 created the TB

3. Investigate sensitivity of model performance to task prompt

2026

• SNUG - Synopsys User Group

2025

• ICLAD - International Conference on LLM-Aided Design
  • attended Hackathon event as well
• DAC - Design Automation Conference

I’ve been looking into reinforcement learning lately to create a racing agent.

As part of this process, I’m reading Reinforcement Learning: An Intro 2nd Edition by Sutton & Barto as my background in ML is supervised/unsupervised learning (blind source separation, variance pretraining). It’s not too useful for my project implementation as I’m experimenting with newer methods like PPO and reward design, but it helps me build a stronger foundation and intuition.

Chapter 5 (Monte Carlo Methods) shows an interesting looking plot of the state-value estimation of an arbitrary player strategy in BlackJack. The plot tells us what the expected return is for a given state under the following strategy: player hits until >= 20 and dealer hits til >= 17.

I wanted to try and recreate this from scratch in Python out of curiosity. Pdfs of the book float around online, such as here (above fig from pdf pg 130).

More Info on the Environment

Agent env is standard BJ, with rewards {1, -1, 0} for winning, losing and drawing. We assume an infinite deck, and the agent’s observation space is their current hand value, whether they have a usable ace, and the visible dealer card. The environment auto hits the player to > 11 so we can’t have > 1 usable ace (if you were wondering why usable ace is binary).

This gives us an observation space of 200 discrete states (Player hand value [12, 21] = 10 states, Dealer Card [A, 10] = 10 states, Usable Ace {0, 1} = 2 states). The player can hit or stand which is a discrete action of 2 states.

The value estimation formula can be found in the core code in the next section, we estimate the value using V(state) = V(s) + (G - V(s))/N where V is the value, G is return from this state to episode end, and N is the total # of times we reached the state.

Code

Core logic to run the MC sim is at core.txt

To get the plots, run sweep.txt (you’ll need core as well, rename both to .py files)

Results

The following plot shows the value estimation improving from 1K to 50K episodes. The usable ace cases are rarer so it takes a bit longer for the values to converge.

Here’s what it looks like after 500K episodes (much closer to the book image than the 50K episode one from above).

Notice how most of the plot is dark blue/purple? That means we’re expected to lose at that point since our strategy of hitting til 20 is suboptimal compared to the dealer one. Let’s try playing the same strategy as the dealer instead.

Much better, the brighter colors (higher values) show that our new strategy doesn’t lose as hard as before.

This post details how using native Pandas operations gives significant speedup over a naive implementation in the context of permuting price paths.

Motivation

Lately, I’ve been looking into how to perform more rigorous backtests for my trading indicators.

The main source of my study is Permutation and Randomization Tests for Trading System Development by Timothy Masters. The book details methodologies for estimating strategy performance such as permuting existing price data to form multiple alternate paths, allowing us to run our strategy on multiple possible realities.

Suppose we want to generate one of these walks for our asset \(S\) from time \(t_0\) to \(t_n\) which moves as \(S_0, S_1, ..., S_n\). \(S_i\) can be Microsoft’s closing price at day \(i\) as an example.

The movement of the asset can be modelled as a small multiplicative modifier \(\sigma\) where \(S_0\cdot\sigma_0=S_1\) and \(S_0\cdot\sigma_0\cdot\sigma_1=S_1\cdot\sigma_1=S_2\) (assuming \(\sigma\) comes from a normal dist. gives us a geometric brownian motion, handy for basic pricing + modelling).

The rest of this post assumes that we work w/ log pricing as this makes the movement additive and gives the benefit of normalizing price differences over long horizons.

Under log pricing, our new equation for asset movement is then \(S_n = S_0 + \sigma_0 + \sigma_1 + ... + \sigma_{n-1}\). Notice how reordering the \(\sigma\)’s doesn’t change the final price as the movements are commutative.

We can obtain \(S_3\) from (\(S_0 + \sigma_0 + \sigma_2 + \sigma_1\)) or (\(S_0 + \sigma_2 + \sigma_1 + \sigma_0\)) despite them being separate paths.

As such, we can generate nearly any number of alternate paths that start at \(S_0\) and end at \(S_n\) by reordering \(\sigma\). The book details why we want to do this, but as a TL:DR, path permutation destroys patterns and preserves trends.

Naive Implementation

Here’s a basic implementation of the path permutation algorithm in Python.

Note 3 key differences in the implementation vs the explanation:

• the Close, High and Low are moved relative to the daily Open to preserve intraday trends
• we permute Open -> Close -> Open to preserve interday trends rather than Close -> Close
• we permute interday and intraday movements separately to break up predictable patterns

def get_example_data():
    import yfinance as yf
    df = yf.download(['MSFT'], period='10y', auto_adjust=True)
    df.columns = ['Close', 'High', 'Low', 'Open', 'Volume']
    return df

def get_perm_slow(log_df, seed=None):
    log_perm_df = np.log(copy.copy(log_df))
    # intraday relative movements of high, low, close relative to open
    # keep first bar constant or else first and last bar won't be identical
    intra_h_o = (log_df['High'] - log_df['Open'])[1:]
    intra_l_o = (log_df['Low'] - log_df['Open'])[1:]
    intra_c_o = (log_df['Close'] - log_df['Open'])[1:]
    # shift(-1) is next row val, shift(1) is prev
    inter_c_o = (log_df['Open'].shift(-1) - log_df['Close'])[:-1]

    # keep price of first bar stable
    np.random.seed(seed)
    perm_ids = np.random.permutation(list(range(0, len(log_df)-1)))
    perm_ids2 = np.random.permutation(perm_ids)
    
    log_perm_df = pd.DataFrame(columns=['Close', 'High', 'Low', 'Open'])
    log_perm_df.loc[0] = copy.copy(log_df.iloc[0])
    
    # starting from first bar
    for idx in range(len(log_df)-1):
        pidx = perm_ids[idx]
        pidx2 = perm_ids2[idx]
        # new open = last Close + permuted inter-bar movement
        o = log_perm_df['Close'].iloc[-1] + inter_c_o.iloc[pidx]
    
        # h,l,c = new open + permuted log inter diff
        h = o + intra_h_o.iloc[pidx2]
        l = o + intra_l_o.iloc[pidx2]
        c = o + intra_c_o.iloc[pidx2]
    
        log_perm_df.loc[len(log_perm_df)] = [c, h, l, o]
    
    log_perm_df.index = log_df.index
    return log_perm_df

It gets the job done, but runs fairly slowly averaging .51 seconds per call on a 10 year chunk of OHLC data.

Back to the drawing board

We want this code to run thousands of times per strategy, meaning we want to squeeze more performance. The bottleneck is the for loop recreating the price path by going through each movement in a random order.

We can greatly expedite this process if we can find a way to write that loop using Pandas operations, and letting the library handle the optimization.

The for loop can be broken down into two operations:

• get a new Open by adding interday movement from the last Close
• get High, Low and Close by adding intraday movement relative to Open

Careful reading of the code + algorithm shows that we can perform these updates independently w/ vectorized calls as long as we take care to propagate the Close -> Open shift properly.

High and Low are intraday movement relative to Open and rely on Close, so we can trivially vectorize their movement relative to the Open.

# Old
for idx in range(len(log_df)-1):
    pidx2 = perm_ids2[idx]
    h = o + intra_h_o.iloc[pidx2]

# New
x['High'] += intra_h_o.iloc[perm_ids2].to_numpy()

Open only depends on the intraday movement from the last Close so we don’t need to modify it.

Intraday Close movement is trickier and requires 2 vectorized operations:

• Add the permuted movement of Close relative to Open as a cumulative sum (each Close movement permanently affects every subsequent Close)
• propagate this movement to every subsequent bar except Close (intraday Close update lags by 1 timestep to every other bar)

Lastly, we add the interday movement from Close to Open using the same trivial vectorization as the High and Low bar, except we use a cumulative sum to propagate the movement to subsequent rows.

Better Implementation

Putting all of this together gives us the following code.

def get_perm(log_df, seed=None):
    np.random.seed(seed)
    # decouple intraday bar from interday shift
    perm_ids = np.random.permutation(list(range(0, len(log_df)-1)))
    perm_ids2 = np.insert(np.random.permutation(perm_ids) + 1, 0, 0)
    
    x = pd.DataFrame(0.0, index=range(len(log_df)), columns=['Close', 'High', 'Low', 'Open'])
    x.index = log_df.index

    intra_h_o = (log_df['High'] - log_df['Open'])
    intra_l_o = (log_df['Low'] - log_df['Open'])
    intra_c_o = (log_df['Close'] - log_df['Open'])
    inter_c_o = (log_df['Open'].shift(-1) - log_df['Close'])[:-1]

    #  Don't move first bar
    intra_h_o.iloc[0] = 0
    intra_l_o.iloc[0] = 0
    intra_c_o.iloc[0] = 0

    start = log_df['Close'].iloc[0]
    x += start
    
    x.loc[x.index[0], :] = x.loc[x.index[0], :].combine_first(log_df.iloc[0])
    
    # Intraday permute
    x['High'] += intra_h_o.iloc[perm_ids2].to_numpy()
    x['Low']  += intra_l_o.iloc[perm_ids2].to_numpy()
    x['Close'] += intra_c_o.iloc[perm_ids2].to_numpy().cumsum()

    # Propagate close movement to future bars
    propagate_close = np.concatenate([[0], intra_c_o.iloc[perm_ids2].cumsum()[:-1]])

    # Already updated Close, other bars lag update by 1
    cols_to_update = x.columns.difference(['Close'])
    x.iloc[1:, x.columns.get_indexer(cols_to_update)] += propagate_close[1:, None]

    # Interday permute
    x = x.add(np.concatenate([[0], np.cumsum(inter_c_o)]), axis=0)

    return x

We’ve replaced the for loop with vectorized operations allowing Pandas to optimize the code using its backend framework (NumPy).

In my testing, the faster implementation gives > 200% speedup, and is able to run the example in ~.002 seconds rather than 0.51.

An example of some permuted Close paths for the last 10 years of MSFT is shown below.