I recently came across this concept when designing a circuit board and ordering a new pair of glasses. In both cases, I was surprised by how streamlined the experience was, the quality of the output and how similar the websites were to each other.

JLCPCB Experience

I designed the circuit board w/ the help of Opus 4.8 and some external plugins and sent the design files to JLCPCB to manufacture. The design is simple, consisting of wiring up five slide pots to two ADCs in a form factor that can sit over a rpi 5 with an audio hat attached.

Front Preview

Back Preview

There were a couple hiccups during the design process related to identifying component dimensions, running the proper router and fixing DRCs, but there were no major issues that took over a day to solve. I tested the assembled PCB after it arrived and it seats and works perfectly too.

Zenni Experience

The glasses were a much simpler process, only requiring me to request my prescription from my optometrist (w/ PD) and enter the measurements into Zenni (a made to order glasses website) to manufacture a pair. I looked through a couple different styles and just picked a random one that caught my eye, image below.

Process Similarities - PCB + Glasses

Cost
I can’t price in the PCB since I don’t know any alternatives, but the glasses were pretty cheap. They cost me $28 without insurance and the PCB run cost $40 for 2 assembled + 3 PCB only. For comparison, my old glasses consistently cost ~$300. Damn.
I think there’s a real argument to be made that glasses from an optometrist tend to be costlier due to name brands, multiple high quality coatings and thin lenses. However, the Zenni’s work perfectly and I feel that a large amount of the cost traditionally comes from the available in-person choices forcing consumers towards pricier choices rather than addressing their needs.

Manufacturing location
Both are made in China. I saw a video recently about sendcutsend, a website that does small batch orders for machining and they mentioned that most metal shops in the US won’t even consider taking smaller orders. US manufacturing tends to focus on huge volume + low margin (mass production), or huge profit + low volume (boutique).
I think this phenomenon comes from the high operating costs in the US, combined with the lack of easy supplier access compared to other countries such as China. Until those issues are addressed, it’ll be rare to see direct to consumer fabrication located w/in the US.
Despite the geographical distance, both products arrived in ~1.5 weeks which I feel is standard for glasses anyways, even from an optometrist.

Ease of Use
Both websites try to make the product comparison, evaluation, and ordering process easy.
JLCPCB’s website expects you to know what you’re doing a bit more compared to Zenni, but it’s very simple to get something working (different from use proficiently) after watching some videos and reading their help guide.

Manufacturing Transparency
This was my favorite part of the process and what inspired me to even document my thoughts.
After sending in the manufacturing order, both websites will show you a pipeline of what stage your order is in (grinding lenses, placing parts on the PCB, moving between warehouses). It’s really fun to look at and I recall checking on the JLCPCB page multiple times a day while I was waiting for my parts to arrive.

Future Challenges

Having used both of these sites, I think that the biggest challenge to making more and better direct to consumer products is a lack of consumer confidence. For both cases, it still felt like a bit of a leap of faith that whatever I ordered was going to work.

I think the PCB case is more understandable as this is my first venture into the field and more experienced designers would have ran a comprehensive test suite and more checks. However, I think it’s a real issue that I was unsure of what size of glasses to buy + how accurate the style was going to be. The AI face measurer + styler felt a bit inaccurate and unrealistic. One potential solution could involve Zenni mailing out a paper copy of the glasses in different sizes, so customers can try them on and see how they’d look.

With that said, both websites were great and I believe that the direct to consumer market will only continue to grow, especially as the need for specialized products at cheaper prices increases. I foresee instilling consumer confidence as the biggest challenge in that regard.

Looking for Dimensions

After breaking them out to a breadboard and validating my initial project design, the next step required finding their dimensions for my PCB design. After looking for a while, I couldn’t find any link to a datasheet on Amazon even after reviewing the original page and looking for similar components.

An online search for similar 75mm B103 slide pots yielded a near-identical looking component with a wiring + dimensioning diagram at the following link. Some info on the page is wrong, such as the travel being 75mm when the length is 75mm and the travel is 60mm (diagram is right, table is wrong).

Measurements + Usage

Just in case, I bought a cheap pair of Amazon calipers and averaged measurements from the 10 pots I bought, and found that there were only 2 notable differences. To use the measurements for your own project, I recommend using the manufacturer spec + my mounting lug dimensions.

• I had the measurements for the 2 center mounting lug positions (they only use the pins)
• The distance between the top and bottom pins are ~1mm longer from my measurements (70.8mm vs 70mm)

The diagrams of my measurements vs the vendor’s are below.

Below is a wiring diagram for a single channel of the pot (view from bottom, flipped vs measurement diagrams which are viewed from above).

They defined the task by scraping 200 github repos based off some heuristics, generating tests using a synthetic pipeline, then feeding the golden compiled program and the repo’s documentation to mini-swe-agent and having it generate the repo + a build script. The build script + code is then ran against the synthetic TB.

Notably, they found that no runs resulted in a 100% TB pass although a couple runs came close. I had some thoughts on how this experiment was carried out and wanted to dive into them in this post.

A full recreation of the repo w/ the necessary modifications to run my experiments can be found at my fork. I’ve added support to make single runs w/ Openrouter as a provider.

• main repo is eval only, I’ve added the steps + extra code to pull the TBs + run the task

Fast run through of paper

Following images are from the paper.

Tasks were created by scraping Github w/ some heuristics to create testbenches. TBs were made in Python using mini-swe-agent w/ Sonnet 4.5 (interesting choice given they had access to better models).

The hand icon is mini-swe-agent’s logo and refers to an agentic step. mini-swe-agent is a lightweight single agent pipeline/flow, comparative to claude-agent-sdk.

The authors have provided docker images per task, and run the agent in there w/ network access disabled along w/ some additional checks to prevent cheating.

Thoughts on Methodology

None of these ideas were created/assisted by AI, they occured as I was going through the paper. I agree with a lot of the methodology and I think the work is novel, I just had some points I was interested in.

1 - task formulation

The goal of the paper is “to measure the ability of software engineering agents to develop software holisitically”. Having the golden executable when creating software seems like a bit of a stretch.

I see the reasoning for it, as we lack a clean specification, but this doesn’t seem to match any software development task I’ve seen outside of reverse engineering and the paper targets “holistic” software development.

2 - questionable test quality from synthetic pipeline

Due to the scale of the benchmark in comparison to the size of the research team, I see the use of a synthetic pipeline.

However, agents can be pretty awful at writing good integration tests w/o human oversight and we can see this in the repo. Cmatrix (one of the tested libs) has very few tests checking the actual animation functionality of the library and a majority of the tests are unit tests checking CLI arg behavior. What tests do exist for animation just check that something happened.

An executable that produces an unsatisfactory animation, but handles the CLI args properly would score extremely high on the benchmark (and could even pass it).

This issue is fundamental to synthetic pipelines being used to create TBs. Without human oversight, it’s very hard to describe the use of software. It’s trivially easy for the model to enumerate the flags and create combinations and write hundreds of garbage tests.

3 - lack of replication

Each of the 200 tasks was ran w/ each of the 9 LMs once. LMs are non-deterministic and I believe that output quality varies more as turns increase (variance of a random walk).

It’s standard practice to use pass@k or make multiple runs to capture output variability and I don’t see a real reason that wasn’t done here.

I could see an argument for cost?, but less LMs or tasks could have been selected.

4 - lack of open source models

self-explanatory

5 - model memorization

Training on github repos is standard practice for LMs now and it’s made model/agent evaluation very difficult. There’s a real possibility of contamination here, especially w/ the newer models.

I’ll come back to this point in my experiment.

My experiment

So where does this leave us?

I have two goals I want to pursue from here:

1. Recreate the experiment and note any learnings
2. Improve on the methodology (multiple ideas I’m saving for a future post)

To do this, I made a fork that runs tasks w/ OpenRouter as the provider. I used deepseek-v4-flash because I don’t have a superintelligence sized budget (the 14 runs I made cost me ~4 buckaroos), but I’m planning to test more down the line.

I selected 3 C repos (cmatrix, bedtools2, doxygen) that most models did well on to recreate the runs for, because I’m the most competent w/ C out of the compilable languages the authors selected.

I used mini-swe-agent w/ the same sys prompt from the repo, and used the following task msg since I couldn’t find the original.

Write some tests for the existing executable, then write your own code and check if it passes. Complete once done by issung COMPLETE_TASK. If a binary errors with Error opening terminal or similar, it's a curses app — focus on --help, version, and error-path testing rather than trying to run it interactively. Use script -qc only if a specific test requires the binary to start successfully under a pty.

Experiment Results + Findings

Deepseek values are from my runs, other values are from the paper/website.

Note that pass rate is test pass %. No runs resulted in a 100% TB test, they all get partial scores.

Cmatrix

I noticed that deepseek outputted the original authors in the headers of the code for 1 of 4 runs. Compared to the next 2 tasks, all the models did well leading me to believe that the models strongly memorized the original code, or the TB is trivial because the application is GUI focused and hard to test.

Runs 2-4 gave 99% TB pass rate (the code is different). I wonder if this is related to the provider (Novita through OpenRouter) potentially caching some of my calls.

Bedtools2 + Doxygen

0% deepseek runs are compile failures due to buggy code.

We can see that multiple runs of deepseek gave a big spread. I’m curious to see what the spread of the closed source models looks like.

Next Steps

1. Improve on methodology

2. Does the model creating the TB matter? Maybe Anthropic models performed a bit better here because Sonnet 4.5 created the TB

3. Investigate sensitivity of model performance to task prompt

2026

• SNUG - Synopsys User Group
• Coreweave - Fully Connected 2026

2025

• ICLAD - International Conference on LLM-Aided Design
  • attended Hackathon event as well
• DAC - Design Automation Conference

I’ve been looking into reinforcement learning lately to create a racing agent.

As part of this process, I’m reading Reinforcement Learning: An Intro 2nd Edition by Sutton & Barto as my background in ML is supervised/unsupervised learning (blind source separation, variance pretraining). It’s not too useful for my project implementation as I’m experimenting with newer methods like PPO and reward design, but it helps me build a stronger foundation and intuition.

Chapter 5 (Monte Carlo Methods) shows an interesting looking plot of the state-value estimation of an arbitrary player strategy in BlackJack. The plot tells us what the expected return is for a given state under the following strategy: player hits until >= 20 and dealer hits til >= 17.

I wanted to try and recreate this from scratch in Python out of curiosity. Pdfs of the book float around online, such as here (above fig from pdf pg 130).

More Info on the Environment

Agent env is standard BJ, with rewards {1, -1, 0} for winning, losing and drawing. We assume an infinite deck, and the agent’s observation space is their current hand value, whether they have a usable ace, and the visible dealer card. The environment auto hits the player to > 11 so we can’t have > 1 usable ace (if you were wondering why usable ace is binary).

This gives us an observation space of 200 discrete states (Player hand value [12, 21] = 10 states, Dealer Card [A, 10] = 10 states, Usable Ace {0, 1} = 2 states). The player can hit or stand which is a discrete action of 2 states.

The value estimation formula can be found in the core code in the next section, we estimate the value using V(state) = V(s) + (G - V(s))/N where V is the value, G is return from this state to episode end, and N is the total # of times we reached the state.

Code

Core logic to run the MC sim is at core.txt

To get the plots, run sweep.txt (you’ll need core as well, rename both to .py files)

Results

The following plot shows the value estimation improving from 1K to 50K episodes. The usable ace cases are rarer so it takes a bit longer for the values to converge.

Here’s what it looks like after 500K episodes (much closer to the book image than the 50K episode one from above).

Notice how most of the plot is dark blue/purple? That means we’re expected to lose at that point since our strategy of hitting til 20 is suboptimal compared to the dealer one. Let’s try playing the same strategy as the dealer instead.

Much better, the brighter colors (higher values) show that our new strategy doesn’t lose as hard as before.

This post details how using native Pandas operations gives significant speedup over a naive implementation in the context of permuting price paths.

Motivation

Lately, I’ve been looking into how to perform more rigorous backtests for my trading indicators.

The main source of my study is Permutation and Randomization Tests for Trading System Development by Timothy Masters. The book details methodologies for estimating strategy performance such as permuting existing price data to form multiple alternate paths, allowing us to run our strategy on multiple possible realities.

Suppose we want to generate one of these walks for our asset \(S\) from time \(t_0\) to \(t_n\) which moves as \(S_0, S_1, ..., S_n\). \(S_i\) can be Microsoft’s closing price at day \(i\) as an example.

The movement of the asset can be modelled as a small multiplicative modifier \(\sigma\) where \(S_0\cdot\sigma_0=S_1\) and \(S_0\cdot\sigma_0\cdot\sigma_1=S_1\cdot\sigma_1=S_2\) (assuming \(\sigma\) comes from a normal dist. gives us a geometric brownian motion, handy for basic pricing + modelling).

The rest of this post assumes that we work w/ log pricing as this makes the movement additive and gives the benefit of normalizing price differences over long horizons.

Under log pricing, our new equation for asset movement is then \(S_n = S_0 + \sigma_0 + \sigma_1 + ... + \sigma_{n-1}\). Notice how reordering the \(\sigma\)’s doesn’t change the final price as the movements are commutative.

We can obtain \(S_3\) from (\(S_0 + \sigma_0 + \sigma_2 + \sigma_1\)) or (\(S_0 + \sigma_2 + \sigma_1 + \sigma_0\)) despite them being separate paths.

As such, we can generate nearly any number of alternate paths that start at \(S_0\) and end at \(S_n\) by reordering \(\sigma\). The book details why we want to do this, but as a TL:DR, path permutation destroys patterns and preserves trends.

Naive Implementation

Here’s a basic implementation of the path permutation algorithm in Python.

Note 3 key differences in the implementation vs the explanation:

• the Close, High and Low are moved relative to the daily Open to preserve intraday trends
• we permute Open -> Close -> Open to preserve interday trends rather than Close -> Close
• we permute interday and intraday movements separately to break up predictable patterns

def get_example_data():
    import yfinance as yf
    df = yf.download(['MSFT'], period='10y', auto_adjust=True)
    df.columns = ['Close', 'High', 'Low', 'Open', 'Volume']
    return df

def get_perm_slow(log_df, seed=None):
    log_perm_df = np.log(copy.copy(log_df))
    # intraday relative movements of high, low, close relative to open
    # keep first bar constant or else first and last bar won't be identical
    intra_h_o = (log_df['High'] - log_df['Open'])[1:]
    intra_l_o = (log_df['Low'] - log_df['Open'])[1:]
    intra_c_o = (log_df['Close'] - log_df['Open'])[1:]
    # shift(-1) is next row val, shift(1) is prev
    inter_c_o = (log_df['Open'].shift(-1) - log_df['Close'])[:-1]

    # keep price of first bar stable
    np.random.seed(seed)
    perm_ids = np.random.permutation(list(range(0, len(log_df)-1)))
    perm_ids2 = np.random.permutation(perm_ids)
    
    log_perm_df = pd.DataFrame(columns=['Close', 'High', 'Low', 'Open'])
    log_perm_df.loc[0] = copy.copy(log_df.iloc[0])
    
    # starting from first bar
    for idx in range(len(log_df)-1):
        pidx = perm_ids[idx]
        pidx2 = perm_ids2[idx]
        # new open = last Close + permuted inter-bar movement
        o = log_perm_df['Close'].iloc[-1] + inter_c_o.iloc[pidx]
    
        # h,l,c = new open + permuted log inter diff
        h = o + intra_h_o.iloc[pidx2]
        l = o + intra_l_o.iloc[pidx2]
        c = o + intra_c_o.iloc[pidx2]
    
        log_perm_df.loc[len(log_perm_df)] = [c, h, l, o]
    
    log_perm_df.index = log_df.index
    return log_perm_df

It gets the job done, but runs fairly slowly averaging .51 seconds per call on a 10 year chunk of OHLC data.

Back to the drawing board

We want this code to run thousands of times per strategy, meaning we want to squeeze more performance. The bottleneck is the for loop recreating the price path by going through each movement in a random order.

We can greatly expedite this process if we can find a way to write that loop using Pandas operations, and letting the library handle the optimization.

The for loop can be broken down into two operations:

• get a new Open by adding interday movement from the last Close
• get High, Low and Close by adding intraday movement relative to Open

Careful reading of the code + algorithm shows that we can perform these updates independently w/ vectorized calls as long as we take care to propagate the Close -> Open shift properly.

High and Low are intraday movement relative to Open and rely on Close, so we can trivially vectorize their movement relative to the Open.

# Old
for idx in range(len(log_df)-1):
    pidx2 = perm_ids2[idx]
    h = o + intra_h_o.iloc[pidx2]

# New
x['High'] += intra_h_o.iloc[perm_ids2].to_numpy()

Open only depends on the intraday movement from the last Close so we don’t need to modify it.

Intraday Close movement is trickier and requires 2 vectorized operations:

• Add the permuted movement of Close relative to Open as a cumulative sum (each Close movement permanently affects every subsequent Close)
• propagate this movement to every subsequent bar except Close (intraday Close update lags by 1 timestep to every other bar)

Lastly, we add the interday movement from Close to Open using the same trivial vectorization as the High and Low bar, except we use a cumulative sum to propagate the movement to subsequent rows.

Better Implementation

Putting all of this together gives us the following code.

def get_perm(log_df, seed=None):
    np.random.seed(seed)
    # decouple intraday bar from interday shift
    perm_ids = np.random.permutation(list(range(0, len(log_df)-1)))
    perm_ids2 = np.insert(np.random.permutation(perm_ids) + 1, 0, 0)
    
    x = pd.DataFrame(0.0, index=range(len(log_df)), columns=['Close', 'High', 'Low', 'Open'])
    x.index = log_df.index

    intra_h_o = (log_df['High'] - log_df['Open'])
    intra_l_o = (log_df['Low'] - log_df['Open'])
    intra_c_o = (log_df['Close'] - log_df['Open'])
    inter_c_o = (log_df['Open'].shift(-1) - log_df['Close'])[:-1]

    #  Don't move first bar
    intra_h_o.iloc[0] = 0
    intra_l_o.iloc[0] = 0
    intra_c_o.iloc[0] = 0

    start = log_df['Close'].iloc[0]
    x += start
    
    x.loc[x.index[0], :] = x.loc[x.index[0], :].combine_first(log_df.iloc[0])
    
    # Intraday permute
    x['High'] += intra_h_o.iloc[perm_ids2].to_numpy()
    x['Low']  += intra_l_o.iloc[perm_ids2].to_numpy()
    x['Close'] += intra_c_o.iloc[perm_ids2].to_numpy().cumsum()

    # Propagate close movement to future bars
    propagate_close = np.concatenate([[0], intra_c_o.iloc[perm_ids2].cumsum()[:-1]])

    # Already updated Close, other bars lag update by 1
    cols_to_update = x.columns.difference(['Close'])
    x.iloc[1:, x.columns.get_indexer(cols_to_update)] += propagate_close[1:, None]

    # Interday permute
    x = x.add(np.concatenate([[0], np.cumsum(inter_c_o)]), axis=0)

    return x

We’ve replaced the for loop with vectorized operations allowing Pandas to optimize the code using its backend framework (NumPy).

In my testing, the faster implementation gives > 200% speedup, and is able to run the example in ~.002 seconds rather than 0.51.

An example of some permuted Close paths for the last 10 years of MSFT is shown below.