Lightning Routing Fees Experiment
Setting various fees policies every month and observe the LN node routing
Introduction
My experience with LN nodes started in 2019 with a simple C-Lightning node. Worked well, I learned a lot of things about Lightning Network in general and how to run a BTC/LN node, what advantages and what inconveniences you can have. But that was just the beginning of LN.
In 2020 then I found out Umbrel. So I switched my C-Lightning node into a new Umbrel node. I have now a Gigabyte Brix NUC with Debian OS.
Just to learn more and also to help the plethora of new users that start installing Umbrel but have no idea whatsoever about what is LN and a LN node.
So the best way was to test it, learn how to use it, discover it and then slowly documenting all my steps in simple guides, so I could share my knowledge with new users, hungry for that information and "how to" steps.
So I built up slowly this node, opening channels, studying the peers closely and making my own peers list (also contain the famous René Pickhardt list of ZeroBaseFee nodes).
I am not saying that my procedures, policies, approaches are the best or are even good. Are just my observations and my own conclusions about how to manage a LN node. I am not an expert or a LN coder, or some "guru", just a pleb observing and passing through my own common sense all is happening with my node.
OK, so I joined to some Rings of Fire, LN+ rings, private groups etc and also connect to specific nodes, during 2 years with this Umbrel node. Until I reached a limit of 40 channels, good stable channels and peers. With many I keep direct contact. Is good to help each others (maybe I would make another guide about this).
Then in 2022 I start a secondary node, a private node, without alias, Tor only, not linked in any way to my real identity or any other online identity. This is part of another experiment that is described in this guide.
I am NOT interested AT ALL in "making money" or "gainz" from LN. My goal is to help making LN a very liquid network of payments, so users would start using it day by day and stop using the damn fucking fiat. We will not start having a healthy payment network if we start by robbing each others and trying to clog the network only for few miserable sats.
When LN will be really stable with really good path finding and healthy nodes, many users using it as daily basis for payments (not only stupid re-balancing), then we can talk about raising fees and have slowly building an incentive model for routing nodes.
We are NOT there yet, no matter what others would say that they earned x amount of sats/month. YOU WILL NOT GET THERE!
And remember: the goal is to FUCK THE BANKS, not to fuck each others...
If we start now charging expensive fees to new users that join to LN use, then they will get scared shit and run, will say that this LN is a shity expensive payment system.
Don't scare them, help them to start using this amazing technology. We will have lots of time later to raise fees to a level of maintaining this as an incentive.
CRAZY GREED IS WHAT KILLS INNOVATION.
How it started... #ZeroFeeFebruary
So in Feb 2022 I started, with one of my strong peers, a movement of #ZeroFeeFebruary between many RoF and groups. Went pretty well with 35+ nodes joining forces and set all channels fees to 0/0 (0 base fee, 0 ppm).
My humble Umbrel node started to 2x, 3x, 4x the number of payments routed through. I also closed some small, zombie channels with peers that didn't grow at all in the last 6 months.
This is a very important aspect: peers that didn't grow enough in the last months, are USELESS, could make more damage than good.
So a warning to all noobs with new nodes: If you plan to run a LN with just 2-3 channels and never grow more, expect that your peers will close the channels. Stack more sats and open more LN channels. Funds in LN are not "lost" or "blocked" as many are trying to scare you. No, funds in LN channels ARE FUCKING LIQUIDITY. That means HAVE TO FLOW. Otherwise is totally useless.
So I start by closing the channels with peers that keep high fees. High fees for me are:
Base fee >1 | anything higher than this is stupid and do more damage than good
ppm >150-200 | sometimes the ppm is necessary to push the liquidity on one side or another, when you really need it, cases for merchants that do not want to loop out but just leave the traffic to do the balancing.
So if a peer connected to my node is using standard policy for all his channels like 2 sats base fee / 300 ppm, I will close the channel, no remorse. Or set such high base fee that all traffic will be blocked on that channel until he close it himself.
Also those using intensively the charge-lnd script. Please stop using it if you are a noob! Or at least learn what you have to do with it and how it works.
Is doing more damage than good. The script is designed to disable channels at a certain level of "profitability", automatically. The user not even know that his channels get disabled and that means CLOSED DOORS. That means no more routing in any way through those channels. So when I see a node with many disabled channels, all the time, that is a clear message: he's using the fucking infamous script, or he's connected to many peers using that script, that peer is a "no go".
Liquidity is flowing in both directions, yes is slow, but have patience. Let the LN living thing to grow naturally and flow naturally. If you put barriers in his path of course it will find another ones.
I tried to be reasonable with those peers that are responsive and understand the situation and adjust their fees accordingly. Slowly the greed will be isolated into the outer rims of the LN Galaxy.
The month of February look like this, with my humble node, 70M total liquidity distributed 50/50 and 40 channels:
In total was 995 payments routed with a total routed of 84M sats.
Highest peak was 95 routed payments and 9M sats in total in one day. In average the whole February months was like 20 txs/day with around 3M sats total routed / day. Not bad, for such a small node. I don't have too many big channels, not even one wumbo.
My testing node had: 5 channels over 4M sats, 14 channels between 4M and 1M sats, 12 channels at 1M sats and 9 channels of 500k sats. In total 40 channels (more or less, closing and opening some).
Comparing February with 6 previous months:
Additional steps I did
I reduced the max HTLC amount to 500k sats to all channels
I configured my node to run on hybrid mode - Tor & clearnet. This help path finding when Tor is clogged.
I activate compaction of my channel.db file, this helps responsiveness of your node and less pending HTLCs
I set a maximum number of HTLCs in lnd.conf file to 10
I maintained all peers connected as much is possible. Sometimes gossip through Tor is really erratic and report offline peers even if they aren't. So what I did, was remove from peer list that peer that appear offline and add it again, if is possible using his clearnet URI. After few seconds, peer is back online and channel is ready. You can use Thunderhub for all these steps.
Conclusions
In general this test was a success for me. My main goal was to see the possibility and capacity of this small node in routing and learn from it, in situations with high volume of traffic. I don't want to dox any of my peers, so I will not post any screenshots with stats on specific channels. But I can say that many of them start moving in this month.
Some key points that I observed:
1 - Fee policy
0/0 fees attract a lot of traffic, but is important also to have peers with similar policy or low fees.
2 - Re-balancing
The peers with high fees, just stalled. Some said that many nodes take advantage of 0/0 fees and rebalance for free. No it is not. I didn't notice something like that. Were few movements from peers with higher fees using the channels with 0/0 fee. I think that most of the routing was just following the natural way of finding the cheapest and fastest path, ignoring the nodes with high fees.
Many said that "depleted" channels are not routing and need to be balanced or closed. It is not true! I had many depleted channels, for one day, yes. But after a while they start moving sats on the way around. All depends if your node have enough many channels, with enough liquidity and balanced at the total level, not individual level
When your total inbound and outbound liquidity is almost equal, you have at least 10-20-30 good channels (not dead, not from the outer rim of the LN Galaxy) then your node will route naturally on both sides. When that ratio of in/out is more than 30% unbalanced, then yes, you start being in trouble, your node can stall.
All this time, I NEVER re-balance in any way, any channel. I didn't use any automated script for that, not even manually. I just sit back and watch.
I also wanted to see if just adjusting slightly the HTLC can help in auto-balancing the channel by redirecting the flow where is needed and there is enough liquid.
3 - Resources, memory load
I notice more memory eaten up when more HTLCs were in pending. So limiting the total number of pending HTLCs in the lnd.conf file helped a bit. Not sure what caused to have constantly at least 3-4 pending HTLCs for 5-10-15 min. My node? My shity HDD? My connection (I was on hybrid mode all time). My peers? General Tor issues?
I understand the meaning of pending HTLCs, but not for such long time. I wish I could do something about them but I don't have enough knowledge or the right information about how to fix them. Maybe LN needs an improvement in this matter.
4 - Node centrality, growth
Is very important your position in the network, the centrality and connections you have. I posted a list with all LN tools available. Use them, are very good into observing your peers, routes etc. Those peers that are far on the edge of the network will not move anything. But those peers that have connections between many of the RoF and central nodes, will have good movement.
So if you have a fresh node, or even an old node, but you have bad connections... change them. And try not to connect with peers that already have same connections as you. RoF are very good to expand, but when you join in many different rings, but with same players, it doesn't help in any way, sometimes even worse, it creates like a loop that never go out.
Expand your connections to nodes that are not in most of the RoF, be the bridge between them and RoF. Explore the LN peers every time you have time and take nodes, observe new nodes.
Connect with more nodes that are using at least 0 base fee and small ppm fee policy. Here is a huge list maintained by René Pickhardt with nodes “0 base fee” policy.
If you have a node with just 2-3 channels... and do not plan to grow more, you better just shut it down and use a simple LN mobile wallet. You are not helping at all the network, neither yourself. We all know that is hard to stack sats, but nobody is forcing you to run a good node with good liquidity.
My experiment is to see that if with relatively small channels (1-5M sats) can obtain good routing for all. Yes you could have 2-3 big channels of 10-20M sats but that I consider more centrality, concentrating more txs into only one spot. Instead of just 1 x 20M I could have 4 ch x 5M sats each and more connections, providing more connectivity. Yes is preferably to have channels bigger than 3M sats, for many reasons.
5 - Use your liquidity node!
Yes, if you already have a LN node, use it damn it! For payments. Wherever you find a merchant that accept LN, pay it with your node. Not that you are helping the merchants, the network, but you make your node more visible in the network. You are pushing forward the liquidity. That's why is named "liquidity" because is FLUID, have to flow, to move, in order to create something big and wonderful.
If you just sit on your LN node waiting for others to route and you to charge them fees... THAT IS PLAIN STUPID. You are just a leecher.
Here I keep updated a list of awesome places where you can use your LN node.
Next step: #March1ppm
For March I would test the same scenario but moving to 0/1 (0 base fee/1 ppm) only.
Also will set max HTLC to 500k sats to all channels bigger than 1M and 150k to all smaller than 1M sats.
UPDATE 1
After 1 week with max HTLC set to 500k sats, I saw too many failed txs. So I start changing a bit the policy. Every morning during my regular coffee, just look at all previous routed txs and channels and adjust the max HTLC to how much I have on my side. That means if a txs comes to my node, automatically will check that “pipe” if is large enough to go through. When the balaance on my side is bigger than 1M sats, I just set 800-900k sats max HTLC or even bigger if is necessary.
For example this channel, of 1M sats in total, I have on my side only 112 765 sats available. So I will set a max 110 000 sats HTLC (rounded, not have to be exactly), because I cannot forward more than that. So a new payment routed, and is bigger than that automatically will not check this route.
But i usually see that users start using more and more MPP so I don’t think will ever be routed a tx bigger than 1Msats, always will be split in smaller parts.
This process took me 5-10 min every morning, not a big deal, I have few channels, not hundreds, so no need for an automated script to set that max HTLC.
After a week doing this, I notice a more natural routing, with less failed HTLCs.
Another task I do every 4-5h / day is to check if there are “offline” channels, in special those that route more. Sometimes, the gossip announcement fail and shows channels in “offline mode” but in fact are not.
So what I do, is go to Thunderhub - Peers, remove the “dead” peer and add it again. After few seconds the channel is back “online”. If the peer is really not online, the adding process will fail so nothing to do anyway, just try again later.
This task can be done with BoS script to program it to do it every 5h, but for the moment I can do it manually (I prefer like that), I don’t have too many disconnections every day, I have good peers too.
I started March with a channel.db file at 1.3GB. Let's see how big will be in 1 month.
UPDATE 2
15 March I did a compacting database. It was already 2.2GB, took 5h to bring down to 1.2GB. I don’t know how this could affect but after few hours, the node start routing like crazy and reached more than 100 routed txs.
Wasn’t a big amount of sats, just 7M sats routed in 100+ txs. I think it helped more the strategy of adjusting max HTLC for the channels with less liquidity on my side.
From 15 March also I start another strategy: for all channels bigger than 1M sats I set min HTLC 99 sats. I let 1 sat min only for few channels where I still use 0/0 fee policy and want to route tiny payments. Is really moving better…
Look at these few channels. I never balance them. It started 2 with balance on my side and 3 with balance on their side. After a week are perfectly balanced.
I did ABSOLUTELY NOTHING! No scripts, no automated balancing, no fees spent, just adjusting max HTLC and leave 0 base fee and 1 ppm.
Some other nodes start opening channels with my node, I don’t know why, maybe because some auto-pilot scripts found my node as “suitable”. But the thing is that they use high fees… so I just close their channels. I do not want greedy peers. It also affect my routing. When I have peers with high fees, this is what happen… total routed txs are going down. Closed those channels and, see what happen next day? Double number of routed txs.
March Conclusion:
It was pretty good month, with 0/1 fee policy. Also to mention that for all channels bigger than 1M sats I set min HTLC to 9 sats. All the rest of the snall channels remain with min 1 sat.
In total I routed 1779 txs and get 113 sats in fees, with a total of 123,218,850 sats moved around, in both directions, with 43 channels always online. I had 2-3 channels that were dead (operators informed me about their problems and I didn’t close them).
The method of adjusting the max HTLC for each channel according to my side liquidity seems to work pretty well. I notice that some more “dormant” channels are waking up and moved some sats. Some more active channels were having good times, moving pretty good amount of sats.
All depends also about the peers and the peers of your peers. If you have just dormant peers that are just waiting for others to move sats and they do not do any LN payment, then those peers are dead end and you better look to replace them. LN must flow in order to grow.
April 10
For April I raised the ppm fee to 10. Let’s see how is going.
UPDATE 15 April
It was a steady average of 50 txs routed/day. But to mention that I had a lot of restarts of my node, due to some other experiments with LNbits.
Also I am doing some tweaking of my lnd.conf file and watch how is going with force closed channels “plague”.
Here is my lnd.conf customization, until now.
UPDATE 19 April
You can adjust your mx HTLC per channel manually if you do not have hundreds of channels or you can automate it with a script. I prefer to do it manually every morning, drinking a coffee and check my node after a busy night, adjusting only those channels that I consider deserve it.
I really don't see the necessity to do unnecessary obsessive rebalancing all the time. That is "fake" traffic over LN and paying fees for nothing.
Path finding and routing can be increased and made more efficient if the routed payments find the right path, where the channels have more liquidity and act accordingly, like the water. If I "hide" that liquidity, the water will stop flowing on that pipe and will go on others routes.
When you use higher or lowers fees but you hide the liquidity, the tx is still coming but is bounced back and you have more failed routing, that means your node will go down in being "seen" as a good route.
Yes, some “privacy advocates” will say that revealing the balance of a channel with mac HTLC is doxxing your node balance. That’s like hiding behind a tree dodging bullets. Useless. The balance of your node channels can be obtained very easily with many other methods and even on public explorers.
Make good flow of the water and in time you can adjust your fees as you want, important is that the water is flowing continuously.
UPDATE 22 April
I increased some small old channels to more than 2.5M sats. The result after few days is this… number of routed txs per day already passed over 200. Let’s see if is just temporary or is a new trend. But I noticed that more and more people are using LN for regular payments (not just useless rebalancing).
In the last week of April I had a big spike in number of payments routed with a maximum 227 txs. Then it went down to a regular 50-60/day.
In April I had many restarts and changes and that was affecting significantly the amount of txs routed. Also I changed a lot channels (closed and open) and gaining again a good flux of txs takes time.
The fact that I raised the fee ppm to 10, not sure it was affecting the routing.
Patience is the key.
I just saw this tweet from Alex Bossworth and I decided to lower the fees for next month of May to 1ppm (Zero Base Fee Forever). Here I posted an answer for Alex.
I think is more important now to build a stable and cheap payment network than start a “fee race” and fuck each others. I don’t think Alex is living from his node fees…
UPDATE 5 May
This month I will try another “experiment”. Apart from selective ppm fee, between 0 and 10ppm, I will play with max HTLC.
My plan is the following:
set max HTLC to 299k sats, when most of the balance is on my side and the channel is bigger than 2M sats.
set max HTLC to 199k sats, when most of the balance is on my side and the channel is lower than 2M sats
Adjust the max HTLC if the balance is getting closer to that max set, going to 199k and then to 19k if is going lower.
Adjust the max HTLC back in 3 steps if the balance is coming back on my side.
In this way, is not necessary to update all the time the channels (is not recommended) and tunnel specific amount of txs through specific channels.
I hope users will use more and more MPP (multi-part payments) and have a better path finding and faster routes.
UPDATE 22 Sept 2022: René Pickhardt just dropped this amazing article, that more or less have the same conclusion like my experiment:
MAY THE ₿ITCOIN BE WITH YOU!
If you appreciate DarthCoin work, you can send some satoshis to darthcoin@getalby.com or darthcoin@stacker.news or darthcoin.blink.sv
or using Cashu Address darthcoin@minibits.cash
If you do not want to subscribe on substack, all DarthCoin Bitcoin guides are also announced on this dedicated Telegram Channel, for easy search and keep track.
To subscribe on substack, click here:
I would love to learn about node maintenance pratices like sheduled restarts etc. Did you do or known any guide about that topic Darth?
Great guide.
I've seen some very strange behaviour with the Muun wallet which some of my users transfer sats to via my node.
If you open an invoice on Muun, copy it and then close the app on an iPhone (not sure what Android does I don't have one to test with) I can see transactions stall for up to 24 hours! When I initially send the payment with the REST API the invoice just times out. Never returns an error.
Subsequent attempts to pay the same invoice return "payment is in transition" in the error field.
I can't find any way to cancel this type of payment.
This will continue for 24 hours and then fail OR if the user opens the Muun app again it will immediately succeed!
I have only observed this behaviour with Muun. I now have a direct channel to Magneto which is Muun's node, their phone apps seem to be linked by temporary private channels to this node.
What I see is that LND's API