a reflection on a year of operating a filler
for the past year, i've been working at Reservoir as one of the core devs on the main filler that powers relay.link. Reservoir hosts a UI for cross-chain swapping & bridging, a gateway API to send intents to, and currently the main filler that fills these intents.
as of the time of this writing, the filler we operate has settled 7.4 million cross chain intents this year, approximately ~$500 million in USD value. We fill intents across 41 different EVM chains, solana, and bitcoin.
this post won't be discussing the trustless relay protocol, the relay.link UI, or the gateway API. It will strictly be about the filler side of things, what I am most familiar with and feel most confident talking about.
whats a filler?
before we get started i think it's important to establish a definition for a filler as well as some other commonly referred to actors in the context of cross-chain intents.
the most commonly referred to actor in the context of intents is a solver. a solver is someone who creates the onchain route that would fulfill a users intent. a solver usually does not have a financial stake in this route. for example, a solver would create a swap route that goes through various AMM protocols, but it does not route through their own inventory. they don't need to put up any money to do the actual work of solving.
a market maker create solutions for intents by using their own inventory, and take on price risks in the process. for example making a market for a memecoin, which has high LP fees due to the risk associated with holding such a volatile asset.
a filler is similar to a market maker, but they try not take on price risk. usually this is done by using trying to only hold major tokens in inventory and hedging them.
a relayer is usually just a dumb actor who submits transactions on behalf of someone. they usually have something at stake that incentivizes them to submit transactions in a timely manner.
our "filler" does a little bit of everything i just defined, except market making. we sometimes create our own swap routes, other times route through our own inventory, and we always relay our fills ourselves.
i like to just think that we do whatever we need to do to help execute user intents, without adhering super hard to a specific definition :)
now that this debate has begun, time for the actual reflection!
reflection
i considered not writing this because there are some things we do that are considered "alpha", and it's hard to explain honestly what it's like to operate a filler without going into depth on these topics. like efficient ways to rebalance, or creating routes between currencies cross-vm with no clear path between them. but, i feel like there is very little insight into what it's like to be on this side of things, so i thought something would be better than nothing.
i also feel like our filler is finally in a good enough position that I personally feel proud about. it's a rare feeling in programming. obviously there is still a lot more to do and improve on, but our current feature set feels robust and competitive compared to others in the industry. so i thought id take some time to reflect on some of our learnings in the past year, and maybe bring some previously not talked about insights in what it's like to operate a filler. can't leak any alpha though!
-- it's probably not a surprise that the main gist of operating a filler is constantly trying to optimize our cost and speed. optimizing our costs means making it cheaper for us to fill an intent, so in return we can give a user a better rate. speed means we can fill the users intent faster, which is important when incorporating time sensitive operations like swaps or minting.
optimizing costs usually means to lower the cost of having funds on a chain. there is usually a cost associated with this, either directly (difficulty to rebalance), or indirectly (liquidity could be used more efficiently elsewhere). there is a multitude of ways to do this, but that is alpha for you to run your own filler and discover :)
filling fast is about being optimistic. filling fast takes on risks, as there is a re-org risk on the deposit. We like taking an optimistic approach and only wait for finality for larger bridges, so we can fill faster for the average user. this is very important for swaps to work, and we found that our fill times make cross-chain swaps feel like same chain swaps, and others have noticed this too.
after about a year of constant optimization:
- our median time to quote a users intent for a native bridge (+major tokens) is ~100ms, and for cross-chain swap intent combining both an origin and destination swap, it is approximately ~1300ms
- our median settlement time is 4 seconds, usually bottle necked by the time it takes for our fill to get included in blocks.
- our total relay failure rate since we started relaying is ~0.6% and is now down to ~0.4% in the past month. it's hard to totally avoid failures when incorporating swaps and unstable centralized sequencers.
swaps...
out of all of the things we do as a filler, swaps is what i have spent the most time on and will spend the most time talking about. swaps is what has consumed me for the better part of this year. i've been working on our swap solving to compliment and wrap around our bridging of native and major tokens. i went into swaps thinking it would be a quick 20-minute adventure, but it ended up being a lot more complex than i anticipated.
with swaps, we have some core primitive ideas of:
- trade types: exact input, exact output, expected output
- solve types: same-chain, cross-chain
- user execution types: push, pull (permit)
any trade type can be matched with any solve type, and any of those can be matched with any execution type. this gets even more complicated when factoring in relay capital fees, gas fees, slippage, reverts and post-swap execution into the equation.
engineering these flows in a maintainable way took a couple refactors to finally get into a position i feel comfortable with. the main difficulty was creating good abstractions that can have interop between the flows, but not have a negative effect on the maintainability.
consider a same-chain swap into a post-hook NFT mint. there are two execution types for this, either the user executes it themselves, or they use a permit to allow us to pull their funds and execute it on their behalf.
at first it seems simple, just have an abstraction for where the funds come from, and then the rest of the flow is identical. however, if it's occurring via a permit, that means we are paying gas on behalf of the user to execute the transaction. because we need to account for this gas from the users payment, the flow needs to now incorporate the user paying more. we can't subtract it from what they already wanted to pay, because often the execution needs an exact amount of output (like NFT mints). we also need to handle swapping the fee into a major currency we want to be paid in (so we don't get paid in meme coins). we then transfer the fee to us, and still need to do the swap + post-swap execution for the user in the same transaction. at this point it's a dramatically different flow!
one thing that helped us tremendously was simulation environments like supersim, which we use to fork multiple mainnets and run thousands of tests against (6 mins to run them all). these tests help us ensure our flows are doing exactly what we want them to do, and any changes we do don't break them. it's hard to account for everything possible, and we have run into some interesting unaccounted for flows in the wild (im looking at you smart accounts).
i think native to native bridging is radically simple to achieve and do well, but incorporating swaps and arbitrary execution takes a problem from 1x complexity into 10x. it's been very fun to get us to the point we are at today and none of it would be possible without the other amazing people we have on our team. it's always rewarding seeing the interesting things our partners are doing with the flows we enable, one of my favorites is what one balance is doing with resource locks.
another interesting problem is the trade type that specifies if the quote amount is denominated in the input or output currency: EXACT_INPUT/EXACT_OUTPUT.
EXACT_INPUT refers to a traditional quote, where a user specifies in their intent how much of a currency they want to sell and want to know how much of the output currency they receive.
EXACT_OUTPUT is when a user wants to know how much of the input currency they need to sell so they can get exactly a specific amount of the output currency.
The problem: Not all DEXs support EXACT_OUTPUT, and swap aggregators are moving away from it as well. But, we need to maintain support for EXACT_OUTPUT because it is crucial for post-swap execution, for example an NFT mint on zora that requires an erc20.
we support EXACT_OUTPUT by still using EXACT_INPUT, but structuring the swap so that the bottom-price of the slippage range is the requested EXACT_OUTPUT amount. So the worst-case slippage scenario will fetch the user their requested output amount.
the way we generate the quote using EXACT_INPUT is straight forward but a bit hacky. we initially flip the swap direction and feed the requested output amount as the new input amount. this gets us an output amount denominated in the input currency, which we can use as a starting point to figure out how much input we will need. we flip the currencies to their normal state, quote again with this indicative amount, and continue to adjust it until we reach an amount out thats within a range we are okay with. its hacky, but it works. we are working with Propeller Heads to use Tycho to help us optimize this further.
out of all of the mistakes i've made this year while working on the filler, the most top of mind one is falling victim to scope and complexity creep. swaps is a perfect example of this, and it introduces problems when integrators are relying on a new feature for their product, but it ends up needing twice as much work as expected. i've gotten a lot better at this, and try to be as generous as possible when giving time estimates to partners.
another more personal learning i've had this year is to avoid making unverified assumptions about solutions. many of the large bugs i've made this year were a result of being optimistic about my thinking and not fully verifying the solution, through proper testing, review, and deep thought. i learned the hard way that assuming things in a complex system does not mesh well with immutable blockchains. one of my biggest personal goals this year was to get out of this habit and instead prefer pessimistic verification, where nothing is assumed unless verified. i've made good progress and have noticed improvements, but i still have more work to do here.
what's next?
as a filler, we are mainly eyeing on adding support for more VMs and allowing anyone to contribute to our inventory in a trustless manner.
at reservoir, we are working on a lot of exciting things, to name a few:
if you have any questions or comments, or want to debate whether or not we are a filler, feel free to DM me on twitter @deor. also, if you are an L2 or working on a new L2 that we currently do not support, DM me so we can support bridging, swapping, and execution between your chain and the other 41 chains we support :)