Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(lnd): mpp #1978

Closed
wants to merge 2 commits into from
Closed

feat(lnd): mpp #1978

wants to merge 2 commits into from

Conversation

sangaman
Copy link
Collaborator

@sangaman sangaman commented Nov 3, 2020

This enables multi path payments with lnd. This involves several significant changes to the swap flow, including a new payReq field added to the SwapReqeuet and SwapAccepted packets. A new test case is added to simulate multipath payments by executing an order between Alice and Bob for an amount that exceeds their direct channel between each other but not the combined balance including their channels with Carol.

Xud as of this PR supports making lnd payments with or without using the invoice payment request string, so payments with older nodes should remain backwards compatible. However, as of lnd 0.12.0 we expect that only payments using the payment request string will work, due to not being able to assign a value to the secret payment address field otherwise.

EDIT by @kilrau : Closes #1975
EDIT by @kilrau : Closes #1934

@sangaman sangaman added the lightning Lightning network & lnd integration label Nov 3, 2020
@kilrau kilrau added this to the 1.3.0 milestone Nov 3, 2020
@kilrau kilrau requested review from a user and raladev November 6, 2020 12:15
@sangaman sangaman force-pushed the lnd/mpp branch 3 times, most recently from c2264be to acfe6f5 Compare November 15, 2020 16:54
Copy link
Contributor

@michael1011 michael1011 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MPP is working just fine on my regtest environment. So it's not XUD's fault that the MPP simulation tests are failing

@kilrau kilrau marked this pull request as ready for review November 17, 2020 15:30
@kilrau
Copy link
Contributor

kilrau commented Nov 17, 2020

As just discussed, please give this some extensive real-world testing @raladev . If this passes your code-level review we can comment out the simulation tests for now and merge as is @erkarl

@raladev
Copy link
Contributor

raladev commented Nov 17, 2020

2_fails_through_thrid_node_maker.log
2_fails_through_thrid_node.log

Does it contain breaking change for xud?

Copy link
Contributor

@raladev raladev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Above

@sangaman
Copy link
Collaborator Author

* It works fine for direct swaps (simple swaps and mpp swaps work fine)

* [ ]  but it does not work with routing node (A -> OzoneYellow -> B) fully; even  simple swap does not work becaouse of no route err.
  [Screenshot from 2020-11-17 19-51-22](https://user-images.githubusercontent.com/29906866/99426097-bfddc900-2914-11eb-922f-3bef30669d2a.png)
  [Screenshot from 2020-11-17 19-51-18](https://user-images.githubusercontent.com/29906866/99426101-c10ef600-2914-11eb-9593-78149e168e13.png)

2_fails_through_thrid_node_maker.log
2_fails_through_thrid_node.log

Does it contain breaking change for xud?

Did you run a similar test for the SendPaymentV2 PR? I'm just wondering to try to narrow down what the cause might be.

There's actually a simulation test that tests multi-hop payments, so it's odd that it would work in the simulation but not in your manual test.

I'm going to try a simulation test that's an mpp using 2 channels between 2 nodes to see if it works.

@ghost
Copy link

ghost commented Nov 18, 2020

utACK from code side

@raladev
Copy link
Contributor

raladev commented Nov 18, 2020

Did you run a similar test for the SendPaymentV2 PR?

Yeap, im sure that multi-hop payments worked fine in v2payments PR

@kilrau
Copy link
Contributor

kilrau commented Dec 14, 2020

Want to prep an intro of how you intend to move forward here and we do a sync call with e.g. @erkarl @michael1011 to go through it and make sure we are not missing anything? @sangaman

@michael1011
Copy link
Contributor

Relevant part of of the latest LND email of Alex Bosworth:

## Payment Addresses Required

Savvy observers may have noticed that in the upcoming LND 0.12.0 release the
payments address feature is set to be required for payments in a
non-backwards-compatible change. Payments made without the address will fail.

This has created some issues due to the fact that the payment address was
previously a hidden field so you couldn't easily specify it even if you wanted
to.

There were some recent merges to return the hidden payment address field in
relevant APIs. In the case of hold invoices, the payment address is still
absent:

"Return payment address when adding hold invoices"
https://github.com/lightningnetwork/lnd/issues/4820

The action item for API users is to capture these payment addresses that will
be present in a future version of LND, and then specify these addresses in
their payment calls.

It should also be noted that the send payment API does not allow you to specify
the payment address when specifying discrete payment details, so if you do
depend on that API, it will no longer function when paying to upgraded
destinations. These kind of changes introduce some new demands of API users so
an issue was created to improve the relevant documentation:

"Document "payment_addr" usage"
https://github.com/lightningnetwork/lnd/issues/4839

This issue also highlights a further complexity in that there is currently no
way for API users to specify their own custom payment addresses, although the
current API documentation may imply that they can.

@sangaman sangaman force-pushed the lnd/mpp branch 4 times, most recently from 7895523 to 1ab79d2 Compare December 18, 2020 15:06
@sangaman
Copy link
Collaborator Author

An update here on my trials & findings:

Currently, when taker pays maker without a direct channel, even without using MPP or specifying max parts, the payment fails with NO_ROUTE. Below is some log output (with max parts specified) from the taker's lnd failing on pathfinding. It successfully found a route with QueryRoutes and you can see at the beginning it says Found route: probability=1, hops=2, fee=1200 mSAT right before initiating the payment.

2020-12-18T06:28:17.3089308Z 2020-12-18 06:27:52.860 [DBG] CRTR: Searching for path to 025f37d60c4739953d412700be23d96356ca089edb324830c3b45e1228cd785ab5, sending 200000000 mSAT
2020-12-18T06:28:17.3091816Z 2020-12-18 06:27:52.861 [DBG] CRTR: Found route: probability=1, hops=2, fee=1200 mSAT
2020-12-18T06:28:17.3093054Z 2020-12-18 06:27:52.862 [DBG] CRTR: Pathfinding perf metrics: nodes=3, edges=5, time=1.28692ms
2020-12-18T06:28:17.3095116Z 2020-12-18 06:27:52.925 [DBG] CRTR: Payment 7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e in state terminate=false, active_shards=0, rem_value=1000000000 mSAT, fee_limit=0 mSAT
2020-12-18T06:28:17.3097592Z 2020-12-18 06:27:52.925 [DBG] CRTR: PaymentSession(7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e): pathfinding for amt=1000000000 mSAT
2020-12-18T06:28:17.3099286Z 2020-12-18 06:27:52.926 [DBG] CRTR: Pathfinding perf metrics: nodes=1, edges=2, time=536.008µs
2020-12-18T06:28:17.3101425Z 2020-12-18 06:27:52.926 [DBG] CRTR: PaymentSession(7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e): pathfinding for amt=500000000 mSAT
2020-12-18T06:28:17.3103072Z 2020-12-18 06:27:52.927 [DBG] CRTR: Pathfinding perf metrics: nodes=1, edges=2, time=996.116µs
2020-12-18T06:28:17.3105060Z 2020-12-18 06:27:52.927 [DBG] CRTR: PaymentSession(7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e): pathfinding for amt=250000000 mSAT
2020-12-18T06:28:17.3106850Z 2020-12-18 06:27:52.928 [DBG] CRTR: Pathfinding perf metrics: nodes=1, edges=2, time=966.816µs
2020-12-18T06:28:17.3108608Z 2020-12-18 06:27:52.928 [DBG] CRTR: PaymentSession(7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e): pathfinding for amt=125000000 mSAT
2020-12-18T06:28:17.3110206Z 2020-12-18 06:27:52.929 [DBG] CRTR: Pathfinding perf metrics: nodes=1, edges=2, time=1.186919ms
2020-12-18T06:28:17.3111956Z 2020-12-18 06:27:52.930 [DBG] CRTR: PaymentSession(7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e): pathfinding for amt=62500000 mSAT
2020-12-18T06:28:17.3113568Z 2020-12-18 06:27:52.932 [DBG] CRTR: Pathfinding perf metrics: nodes=1, edges=2, time=2.019233ms
2020-12-18T06:28:17.3115301Z 2020-12-18 06:27:52.932 [DBG] CRTR: PaymentSession(7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e): pathfinding for amt=31250000 mSAT
2020-12-18T06:28:17.3117312Z 2020-12-18 06:27:52.932 [DBG] CRTR: Pathfinding perf metrics: nodes=1, edges=2, time=332.805µs
2020-12-18T06:28:17.3120034Z 2020-12-18 06:27:52.932 [DBG] CRTR: PaymentSession(7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e): pathfinding for amt=15625000 mSAT
2020-12-18T06:28:17.3122589Z 2020-12-18 06:27:52.932 [DBG] CRTR: Pathfinding perf metrics: nodes=1, edges=2, time=366.006µs
2020-12-18T06:28:17.3125604Z 2020-12-18 06:27:52.933 [DBG] CRTR: PaymentSession(7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e): not splitting because minimum shard amount 10000000 mSAT has been reached
2020-12-18T06:28:17.3129331Z 2020-12-18 06:27:52.933 [WRN] CRTR: Failed to find route for payment 7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e: unable to find a path to destination
2020-12-18T06:28:17.3132278Z 2020-12-18 06:27:52.933 [DBG] CRTR: Marking payment 7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e permanently failed with no route: no_route
2020-12-18T06:28:17.3135104Z 2020-12-18 06:27:52.944 [DBG] CRTR: Payment 7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e in state terminate=true, active_shards=0, rem_value=1000000000 mSAT, fee_limit=0 mSAT
2020-12-18T06:28:17.3137483Z 2020-12-18 06:27:52.944 [ERR] CRTR: Payment with hash 7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e failed: no_route
2020-12-18T06:28:17.3139892Z 2020-12-18 06:27:53.088 [INF] CRTR: Pruning channel graph using block b1f39d5ffdee04a5044c9746f1dcbb767afaa53e79065613e4b944c56844f530 (height=391)

On the maker (receiver) side I also found some odd output where the invoice was seemingly canceled right after being added, I have no explanation for this but you can see it's for the same hash as above. It may be unrelated but it was too odd to ignore.

2020-12-18T06:28:14.5354879Z 2020-12-18 06:27:52.894 [DBG] INVC: Invoice(pay_hash=7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e): added with terms amt=1000000000 mSAT, expiry=1h0m0s, final_cltv_delta=1325
2020-12-18T06:28:14.5359230Z 2020-12-18 06:27:52.895 [DBG] INVC: Adding invoice '7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e' to expiry watcher, expiration: 2020-12-18 07:27:52.893441405 +0000 UTC m=+3638.923453166
2020-12-18T06:28:14.5362749Z 2020-12-18 06:27:52.899 [INF] INVC: New single invoice subscription client: id=4, ref=(pay_hash=7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e)
2020-12-18T06:28:14.5365475Z 2020-12-18 06:27:52.965 [DBG] INVC: Invoice(pay_hash=7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e): canceling invoice
2020-12-18T06:28:14.5368482Z 2020-12-18 06:27:52.966 [DBG] INVC: Invoice(pay_hash=7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e): canceled
2020-12-18T06:28:14.5371079Z 2020-12-18 06:27:52.966 [INF] IRPC: Canceled invoice 7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e
2020-12-18T06:28:14.5372658Z 2020-12-18 06:27:53.040 [DBG] RPCS: [listchannels] fetched 3 channels from DB
2020-12-18T06:28:14.5373718Z 2020-12-18 06:27:53.043 [DBG] RPCS: [listchannels] fetched 3 channels from DB
2020-12-18T06:28:14.5374903Z 2020-12-18 06:27:53.046 [DBG] RPCS: Target sat/kw for closing transaction: 50000
2020-12-18T06:28:14.5377422Z 2020-12-18 06:27:53.047 [DBG] HSWC: Requesting local channel close: peer=03fb2d3622649866646f3ae8e381dfdfdedc55d7be9fb67cd901bd8cf9331b297d@127.0.0.1:38439, chan_id=904d5d2833c4761e59db954f861c9d4fa8e75fee9c9eb3d9cd92454485bb2975
2020-12-18T06:28:14.5380217Z 2020-12-18 06:27:53.047 [INF] PEER: Local close channel request delivered to peer: 03fb2d3622649866646f3ae8e381dfdfdedc55d7be9fb67cd901bd8cf9331b297d
2020-12-18T06:28:14.5382312Z 2020-12-18 06:27:53.050 [INF] PEER: Delivery addr for channel close: sltc1qxs3ygj5ppu7yj7x3c2dpls7yjef04gk7nd5gjf
2020-12-18T06:28:14.5384505Z 2020-12-18 06:27:53.050 [INF] CHCL: Ideal fee for closure of ChannelPoint(7529bb85444592cdd9b39e9cee5fe7a84f9d1c864f95db591e76c433285d4d90:0) is: 36200 sat
2020-12-18T06:28:14.5391348Z 2020-12-18 06:27:53.050 [INF] CHCL: ChannelPoint(7529bb85444592cdd9b39e9cee5fe7a84f9d1c864f95db591e76c433285d4d90:0): initiating shutdown
2020-12-18T06:28:14.5394237Z 2020-12-18 06:27:53.050 [INF] NANN: Announcing channel(7529bb85444592cdd9b39e9cee5fe7a84f9d1c864f95db591e76c433285d4d90:0) disabled [requested]
2020-12-18T06:28:14.5396924Z 2020-12-18 06:27:53.053 [INF] HSWC: Removing channel link with ChannelID(904d5d2833c4761e59db954f861c9d4fa8e75fee9c9eb3d9cd92454485bb2975)

If I change xud code to go back to specifying the hash, destination, amount, and final cltv delta manually rather than the payment request (aka invoice), multi hop payments work again. That won't work with multi path payments since those require the payment address secret which cannot be specified on its own currently, but at least that isolates the problem. I've decoded the payment request to make sure everything looks correct and in fact it matches up with the values we specify independently. Here's an example decoded invoice:

        "tags": [
            {
                "type": "p",
                "length": 52,
                "description": "payment_hash",
                "value": "7f7cd226fe5af07a11d0a483e1ac85dfbbeddcde5e45627536392d1588abb85e"
            },
            {
                "type": "d",
                "length": 0,
                "description": "description",
                "value": ""
            },
            {
                "type": "c",
                "length": 3,
                "description": "min_final_cltv_expiry",
                "value": 1325
            },
            {
                "type": "s",
                "length": 52,
                "description": "payment_secret",
                "value": "2de1a1108dbe5e753101dab6ed87c817834ab8bae417fe78ab38daeebc494213"
            },
            {
                "type": "9",
                "length": 4,
                "description": "feature_bits",
                "value": "00101000001000000000"
            }
        ],

Setting that invoice on the SendPaymentRequest results in multi hop payments failing, but setting the exact same cltv expiry, rhash, and amount independently works. I don't really know why but I suspect it might have something to do with us using hold invoices and relatively large cltv expiry values, those are the only things that really differentiate our use case from the standard lnd invoice multi-hop payments that happen all the time without issues. For example, lncli addholdinvoice doesn't even give an option to set a final cltv expiry.

Next things to try are:

  1. Try the same thing on lnd 0.12 in the hopes that some underlying issue has been resolved in the newer version.

  2. Create a hold invoice in xud and try to pay it with lncli using only the payment request. If this fails as well we can rule out an issue in the way xud is sending lnd payments.

If those don't work I think it needs to be looked into on the lnd side of things because I'm out of ideas from what to change or try in the xud code.

@kilrau
Copy link
Contributor

kilrau commented Dec 22, 2020

Did you try https://github.com/lightningnetwork/lnd/releases/tag/v0.12.0-beta.rc2 ?
Did you try multi-hop invoice payments via lncli with an extended cltv expiry (by trying to pay an invoice from xud manually via lncli). Failed? Opened an issue in https://github.com/lightningnetwork/lnd ?

@sangaman
Copy link
Collaborator Author

I am further mystified now because intercepting a pay req generated by xud and paying it via lncli worked. You can see the output below with the extended cltv expiry for the payment.

simnet > lndltc-lncli payinvoice lnsltc10m1p07ynsspp5paye2mv0l8lanq9vakqjjvau4stevu6jsynr09nmh6n3fcpuxvgsdqqcqrpfdsp5md67v99deyxgskpu4n5pr9heezguk9a5pc3a5mupg5ku2hjg6rgs9qy9qsqdh5eu3kg7lfvxdfr3j7cx0mvgrrntcew6zkm5axnuf323g95gwfxwprc4z05vl5pr2fzpkpjllccla2yvx2wp9z7yl2f5j96xdvf6vcpj8mtqv
Payment hash: 0f49956d8ff9ffd980aced812933bcac17967352812637967bbea714e03c3311
Description: 
Amount (in satoshis): 1000000
Fee limit (in satoshis): 1000000
Destination: 034270e92a79521270352c01520c6c20867ecf448ba4119eaaefa1445377c477d4
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+--------------------+--------------------------------------------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT           | ROUTE                                      |
+------------+--------------+--------------+--------------+-----+----------+--------------------+--------------------------------------------+
| SUCCEEDED  |        0.016 |       78.667 | 1000000      | 2   |   259225 | 282685539012902913 | 0252f366111259996a10->034270e92a7952127035 |
+------------+--------------+--------------+--------------+-----+----------+--------------------+--------------------------------------------+
Amount + fee:   1000000 + 2 sat
Payment hash:   0f49956d8ff9ffd980aced812933bcac17967352812637967bbea714e03c3311
Payment status: SUCCEEDED, preimage: 50c808f4d3226b565c1899dc30a72c69c525d0a24490d02747f0837fb8e5293c
simnet > lndltc-lncli decodepayreq lnsltc10m1p07ynsspp5paye2mv0l8lanq9vakqjjvau4stevu6jsynr09nmh6n3fcpuxvgsdqqcqrpfdsp5md67v99deyxgskpu4n5pr9heezguk9a5pc3a5mupg5ku2hjg6rgs9qy9qsqdh5eu3kg7lfvxdfr3j7cx0mvgrrntcew6zkm5axnuf323g95gwfxwprc4z05vl5pr2fzpkpjllccla2yvx2wp9z7yl2f5j96xdvf6vcpj8mtqv
{
    "destination": "034270e92a79521270352c01520c6c20867ecf448ba4119eaaefa1445377c477d4",
    "payment_hash": "0f49956d8ff9ffd980aced812933bcac17967352812637967bbea714e03c3311",
    "num_satoshis": "1000000",
    "timestamp": "1608666640",
    "expiry": "3600",
    "description": "",
    "description_hash": "",
    "fallback_addr": "",
    "cltv_expiry": "1325",
    "route_hints": [
    ],
    "payment_addr": "db75e614adc90c88583cace81196f9c891cb17b40e23da6f81452dc55e48d0d1",
    "num_msat": "1000000000",
    "features": {
        "9": {
            "name": "tlv-onion",
            "is_required": false,
            "is_known": true
        },
        "15": {
            "name": "payment-addr",
            "is_required": false,
            "is_known": true
        },
        "17": {
            "name": "multi-path-payments",
            "is_required": false,
            "is_known": true
        }
    }
}

You can also see from the payment history that it took two hops

        {
            "payment_hash": "0f49956d8ff9ffd980aced812933bcac17967352812637967bbea714e03c3311",
            "value": "1000000",
            "creation_date": "1608666694",
            "fee": "2",
            "payment_preimage": "50c808f4d3226b565c1899dc30a72c69c525d0a24490d02747f0837fb8e5293c",
            "value_sat": "1000000",
            "value_msat": "1000000000",
            "payment_request": "lnsltc10m1p07ynsspp5paye2mv0l8lanq9vakqjjvau4stevu6jsynr09nmh6n3fcpuxvgsdqqcqrpfdsp5md67v99deyxgskpu4n5pr9heezguk9a5pc3a5mupg5ku2hjg6rgs9qy9qsqdh5eu3kg7lfvxdfr3j7cx0mvgrrntcew6zkm5axnuf323g95gwfxwprc4z05vl5pr2fzpkpjllccla2yvx2wp9z7yl2f5j96xdvf6vcpj8mtqv",
            "status": "SUCCEEDED",
            "fee_sat": "2",
            "fee_msat": "2000",
            "creation_time_ns": "1608666694816338481",
            "htlcs": [
                {
                    "status": "SUCCEEDED",
                    "route": {
                        "total_time_lock": 259225,
                        "total_fees": "2",
                        "total_amt": "1000002",
                        "hops": [
                            {
                                "chan_id": "282685539012902913",
                                "chan_capacity": "40000000",
                                "amt_to_forward": "1000000",
                                "fee": "2",
                                "expiry": 258649,
                                "amt_to_forward_msat": "1000000000",
                                "fee_msat": "2000",
                                "pub_key": "0252f366111259996a101c5a82c880e9c95c8278d7e668c40163c0db656161cbbf",
                                "tlv_payload": true,
                                "mpp_record": null,
                                "custom_records": {
                                }
                            },
                            {
                                "chan_id": "282859261850091520",
                                "chan_capacity": "0",
                                "amt_to_forward": "1000000",
                                "fee": "0",
                                "expiry": 258649,
                                "amt_to_forward_msat": "1000000000",
                                "fee_msat": "0",
                                "pub_key": "034270e92a79521270352c01520c6c20867ecf448ba4119eaaefa1445377c477d4",
                                "tlv_payload": true,
                                "mpp_record": {
                                    "payment_addr": "db75e614adc90c88583cace81196f9c891cb17b40e23da6f81452dc55e48d0d1",
                                    "total_amt_msat": "1000000000"
                                },
                                "custom_records": {
                                }
                            }
                        ],
                        "total_fees_msat": "2000",
                        "total_amt_msat": "1000002000"
                    },
                    "attempt_time_ns": "1608666694833192656",
                    "resolve_time_ns": "1608666773484145707",
                    "failure": null,
                    "preimage": "50c808f4d3226b565c1899dc30a72c69c525d0a24490d02747f0837fb8e5293c"
                }
            ],
            "payment_index": "10",
            "failure_reason": "FAILURE_REASON_NONE"
        }

Lncli is using SendPaymentV2, as far as I can tell I'm doing the exact same thing. I've been trying again to comb through every line of code to see what the difference/problem may be.

@sangaman sangaman force-pushed the lnd/mpp branch 2 times, most recently from 97343bc to c98ef76 Compare December 23, 2020 01:12
@sangaman
Copy link
Collaborator Author

sangaman commented Dec 23, 2020

I finally figured out the culprit. Here's the old code for setting the fee limit.

    const fee = Math.floor(MAX_FEE_RATIO * request.getAmt());
    request.setFeeLimitSat(fee);

When we stop setting the amt field on the send payment request (since amount is specified by the payment request), the fee limit is set to 0 instead of the usual 3% of the total order size. Lnd treats a 0 fee limit on the send payment request as a hard zero fee limit, putting aside the fact that 0 is the default value for an unassigned number field in protobuf. Therefore, it rejects any route including hops (since hops almost always incur at least some small fee), which explains why payments with direct channels were succeeding but multi-hop paths were failing. lncli sets a fee limit of 100 sats by default, even if not specified on the command line, which is why paying an xud invoice with lncli worked above.

I cleaned up the commit history - all tests are passing now and this is ready for real review and testing.

@ghost ghost changed the title WIP: lnd mpp feat(lnd): mpp Dec 23, 2020
@@ -917,6 +937,9 @@ class Swaps extends EventEmitter {
return;
}

// TODO: re-enable add invoice *after* swap accepted once we are able to specify the
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to clean up this TODO in this PR?

@raladev
Copy link
Contributor

raladev commented Dec 23, 2020

  • got failed MPP swap for LTC/BTC Pair

Steps:

  1. as a maker:
    1.1 openchannel BTC 0.2 OzoneYellow 0.1
    1.2 openchannel BTC 0.02 OzoneYellow 0.01
    1.3 openchannel LTC 0.05 OzoneYellow 0.025
    1.4 openchannel LTC 0.2 OzoneYellow 0.1
  2. as a taker:
    2.1 openchannel BTC 0.03 OzoneYellow 0.015
    2.2 openchannel BTC 0.3 OzoneYellow 0.15
    2.3 openchannel LTC 0.25 OzoneYellow 0.12
    2.4 openchannel LTC 0.025 OzoneYellow 0.012
  3. as a maker sell 0.12 LTC/BTC 0.01
  4. asa taker buy 0.12 LTC/BTC mkt

Actual result:
Swap Timeout
Screenshot from 2020-12-23 20-23-08

Notes:

my trading limits and channels after second step:

maker:

simnet > lndbtc-lncli listchannels
{
    "channels": [
        {
            "active": true,
            "remote_pubkey": "0252f366111259996a101c5a82c880e9c95c8278d7e668c40163c0db656161cbbf",
            "channel_point": "be8db5233056dc2a3441d517423e69e83a41d606a25cec0528dcfd77cb512f73:0",
            "chan_id": "284026943198789632",
            "capacity": "2342100",
            "local_balance": "1087126",
            "remote_balance": "1245923",
            "commit_fee": "9051",
            "commit_weight": "724",
            "fee_per_kw": "12500",
            "unsettled_balance": "0",
            "total_satoshis_sent": "2803",
            "total_satoshis_received": "0",
            "num_updates": "6",
            "pending_htlcs": [
            ],
            "csv_delay": 281,
            "private": false,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "23421",
            "remote_chan_reserve_sat": "23421",
            "static_remote_key": true,
            "commitment_type": "STATIC_REMOTE_KEY",
            "lifetime": "9906",
            "uptime": "9906",
            "close_address": "",
            "push_amount_sat": "1243120",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 281,
                "chan_reserve_sat": "23421",
                "dust_limit_sat": "573",
                "max_pending_amt_msat": "2318679000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "remote_constraints": {
                "csv_delay": 281,
                "chan_reserve_sat": "23421",
                "dust_limit_sat": "573",
                "max_pending_amt_msat": "2318679000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            }
        },
        {
            "active": true,
            "remote_pubkey": "0252f366111259996a101c5a82c880e9c95c8278d7e668c40163c0db656161cbbf",
            "channel_point": "4e0818a9d2a1bd195f5c020ea74cf3638900f32dc79962add83e7c40cefd13ed:0",
            "chan_id": "284157785082494976",
            "capacity": "20000000",
            "local_balance": "9990950",
            "remote_balance": "10000000",
            "commit_fee": "9050",
            "commit_weight": "724",
            "fee_per_kw": "12500",
            "unsettled_balance": "0",
            "total_satoshis_sent": "0",
            "total_satoshis_received": "0",
            "num_updates": "0",
            "pending_htlcs": [
            ],
            "csv_delay": 2016,
            "private": false,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "200000",
            "remote_chan_reserve_sat": "200000",
            "static_remote_key": true,
            "commitment_type": "STATIC_REMOTE_KEY",
            "lifetime": "2729",
            "uptime": "2729",
            "close_address": "",
            "push_amount_sat": "10000000",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 2016,
                "chan_reserve_sat": "200000",
                "dust_limit_sat": "573",
                "max_pending_amt_msat": "19800000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "remote_constraints": {
                "csv_delay": 2016,
                "chan_reserve_sat": "200000",
                "dust_limit_sat": "573",
                "max_pending_amt_msat": "19800000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            }
        }
    ]
}
simnet > lndltc-lncli listchannels
{
    "channels": [
        {
            "active": true,
            "remote_pubkey": "0252f366111259996a101c5a82c880e9c95c8278d7e668c40163c0db656161cbbf",
            "channel_point": "30ecdadab1667fc9e24ba8f6de6897fe360a59bd6e01a54062f04a2fcdf0808f:0",
            "chan_id": "284265537222082560",
            "capacity": "20000000",
            "local_balance": "9963800",
            "remote_balance": "10000000",
            "commit_fee": "36200",
            "commit_weight": "724",
            "fee_per_kw": "50000",
            "unsettled_balance": "0",
            "total_satoshis_sent": "0",
            "total_satoshis_received": "0",
            "num_updates": "0",
            "pending_htlcs": [
            ],
            "csv_delay": 576,
            "private": false,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "200000",
            "remote_chan_reserve_sat": "200000",
            "static_remote_key": true,
            "commitment_type": "STATIC_REMOTE_KEY",
            "lifetime": "2823",
            "uptime": "2823",
            "close_address": "",
            "push_amount_sat": "10000000",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 576,
                "chan_reserve_sat": "200000",
                "dust_limit_sat": "54600",
                "max_pending_amt_msat": "19800000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "remote_constraints": {
                "csv_delay": 576,
                "chan_reserve_sat": "200000",
                "dust_limit_sat": "54600",
                "max_pending_amt_msat": "19800000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            }
        },
        {
            "active": true,
            "remote_pubkey": "0252f366111259996a101c5a82c880e9c95c8278d7e668c40163c0db656161cbbf",
            "channel_point": "07456d6f76387d7d79f5f4738ca04c82e965a2443930f12fad160e43c0fd3be8:0",
            "chan_id": "284276532338294784",
            "capacity": "5000000",
            "local_balance": "2463800",
            "remote_balance": "2500000",
            "commit_fee": "36200",
            "commit_weight": "724",
            "fee_per_kw": "50000",
            "unsettled_balance": "0",
            "total_satoshis_sent": "0",
            "total_satoshis_received": "0",
            "num_updates": "0",
            "pending_htlcs": [
            ],
            "csv_delay": 576,
            "private": false,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "54600",
            "remote_chan_reserve_sat": "54600",
            "static_remote_key": true,
            "commitment_type": "STATIC_REMOTE_KEY",
            "lifetime": "2220",
            "uptime": "2220",
            "close_address": "",
            "push_amount_sat": "2500000",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 576,
                "chan_reserve_sat": "54600",
                "dust_limit_sat": "54600",
                "max_pending_amt_msat": "4950000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "remote_constraints": {
                "csv_delay": 576,
                "chan_reserve_sat": "54600",
                "dust_limit_sat": "54600",
                "max_pending_amt_msat": "4950000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            }
        }
    ]
}
simnet > tradinglimits

Trading Limits:
┌──────────┬────────────┬────────────┬──────────────┬───────────────┐
│ Currency │ Max Buy    │ Max Sell   │ Reserved Buy │ Reserved Sell │
├──────────┼────────────┼────────────┼──────────────┼───────────────┤
│ BTC      │ 0.11022502 │ 0.10854655 │ 0            │ 0             │
├──────────┼────────────┼────────────┼──────────────┼───────────────┤
│ ETH      │ 0.1        │ 30         │ 0            │ 0             │
├──────────┼────────────┼────────────┼──────────────┼───────────────┤
│ LTC      │ 0.122454   │ 0.12173    │ 0            │ 0             │

taker:

Trading Limits:
┌──────────┬──────────┬──────────┬──────────────┬───────────────┐
│ Currency │ Max Buy  │ Max Sell │ Reserved Buy │ Reserved Sell │
├──────────┼──────────┼──────────┼──────────────┼───────────────┤
│ BTC      │ 0.1617   │ 0.161519 │ 0            │ 0             │
├──────────┼──────────┼──────────┼──────────────┼───────────────┤
│ ETH      │ 0        │ 15       │ 0            │ 0             │
├──────────┼──────────┼──────────┼──────────────┼───────────────┤
│ LTC      │ 0.128954 │ 0.13923  │ 0            │ 0             │
└──────────┴──────────┴──────────┴──────────────┴───────────────┘
simnet > lndbtc-lncli listchannels
{
    "channels": [
        {
            "active": true,
            "remote_pubkey": "0252f366111259996a101c5a82c880e9c95c8278d7e668c40163c0db656161cbbf",
            "channel_point": "e57505e389f9b31c4db7fe8132e03a54e79de8c331ee69c252db9ed301fdeac1:0",
            "chan_id": "284180874826678272",
            "capacity": "3000000",
            "local_balance": "1490950",
            "remote_balance": "1500000",
            "commit_fee": "9050",
            "commit_weight": "724",
            "fee_per_kw": "12500",
            "unsettled_balance": "0",
            "total_satoshis_sent": "0",
            "total_satoshis_received": "0",
            "num_updates": "0",
            "pending_htlcs": [
            ],
            "csv_delay": 360,
            "private": false,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "30000",
            "remote_chan_reserve_sat": "30000",
            "static_remote_key": true,
            "commitment_type": "STATIC_REMOTE_KEY",
            "lifetime": "1031",
            "uptime": "1027",
            "close_address": "",
            "push_amount_sat": "1500000",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 360,
                "chan_reserve_sat": "30000",
                "dust_limit_sat": "573",
                "max_pending_amt_msat": "2970000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "remote_constraints": {
                "csv_delay": 360,
                "chan_reserve_sat": "30000",
                "dust_limit_sat": "573",
                "max_pending_amt_msat": "2970000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            }
        },
        {
            "active": true,
            "remote_pubkey": "0252f366111259996a101c5a82c880e9c95c8278d7e668c40163c0db656161cbbf",
            "channel_point": "ea83cd9c7979d4b56194fbe12fa0e577f359477f4763db25e5ac1b833537ffcc:0",
            "chan_id": "284170979222028288",
            "capacity": "30000000",
            "local_balance": "14990950",
            "remote_balance": "15000000",
            "commit_fee": "9050",
            "commit_weight": "724",
            "fee_per_kw": "12500",
            "unsettled_balance": "0",
            "total_satoshis_sent": "0",
            "total_satoshis_received": "0",
            "num_updates": "2",
            "pending_htlcs": [
            ],
            "csv_delay": 2016,
            "private": false,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "300000",
            "remote_chan_reserve_sat": "300000",
            "static_remote_key": true,
            "commitment_type": "STATIC_REMOTE_KEY",
            "lifetime": "1031",
            "uptime": "1027",
            "close_address": "",
            "push_amount_sat": "15000000",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 2016,
                "chan_reserve_sat": "300000",
                "dust_limit_sat": "573",
                "max_pending_amt_msat": "29700000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "remote_constraints": {
                "csv_delay": 2016,
                "chan_reserve_sat": "300000",
                "dust_limit_sat": "573",
                "max_pending_amt_msat": "29700000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            }
        }
    ]
}
simnet > lndltc-lncli listchannels
{
    "channels": [
        {
            "active": true,
            "remote_pubkey": "0252f366111259996a101c5a82c880e9c95c8278d7e668c40163c0db656161cbbf",
            "channel_point": "680d22d54331d9bfaf205833f3d2f9159795330b2632f1993e1b207eabdf2c45:0",
            "chan_id": "284282029896433664",
            "capacity": "25000000",
            "local_balance": "12963800",
            "remote_balance": "12000000",
            "commit_fee": "36200",
            "commit_weight": "724",
            "fee_per_kw": "50000",
            "unsettled_balance": "0",
            "total_satoshis_sent": "0",
            "total_satoshis_received": "0",
            "num_updates": "2",
            "pending_htlcs": [
            ],
            "csv_delay": 576,
            "private": false,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "250000",
            "remote_chan_reserve_sat": "250000",
            "static_remote_key": true,
            "commitment_type": "STATIC_REMOTE_KEY",
            "lifetime": "1038",
            "uptime": "1035",
            "close_address": "",
            "push_amount_sat": "12000000",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 576,
                "chan_reserve_sat": "250000",
                "dust_limit_sat": "54600",
                "max_pending_amt_msat": "24750000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "remote_constraints": {
                "csv_delay": 576,
                "chan_reserve_sat": "250000",
                "dust_limit_sat": "54600",
                "max_pending_amt_msat": "24750000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            }
        },
        {
            "active": true,
            "remote_pubkey": "0252f366111259996a101c5a82c880e9c95c8278d7e668c40163c0db656161cbbf",
            "channel_point": "7bbf278d408bf9e6bd4dc9db5516b224dba4736f93e57053ac4b34b80fa7a273:0",
            "chan_id": "284293025012711424",
            "capacity": "2500000",
            "local_balance": "1263800",
            "remote_balance": "1200000",
            "commit_fee": "36200",
            "commit_weight": "724",
            "fee_per_kw": "50000",
            "unsettled_balance": "0",
            "total_satoshis_sent": "0",
            "total_satoshis_received": "0",
            "num_updates": "0",
            "pending_htlcs": [
            ],
            "csv_delay": 576,
            "private": false,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "54600",
            "remote_chan_reserve_sat": "54600",
            "static_remote_key": true,
            "commitment_type": "STATIC_REMOTE_KEY",
            "lifetime": "1038",
            "uptime": "1035",
            "close_address": "",
            "push_amount_sat": "1200000",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 576,
                "chan_reserve_sat": "54600",
                "dust_limit_sat": "54600",
                "max_pending_amt_msat": "2475000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "remote_constraints": {
                "csv_delay": 576,
                "chan_reserve_sat": "54600",
                "dust_limit_sat": "54600",
                "max_pending_amt_msat": "2475000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            }
        }
    ]
}

Copy link
Contributor

@raladev raladev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

above

@kilrau
Copy link
Contributor

kilrau commented Dec 23, 2020

How about regular swaps (fitting into one channel)? @raladev

@raladev
Copy link
Contributor

raladev commented Dec 23, 2020

How about regular swaps (fitting into one channel)? @raladev

no problem with one-channel swaps

Copy link
Contributor

@michael1011 michael1011 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK

@@ -575,6 +591,7 @@ class LndClient extends SwapClient {
};

public sendSmallestAmount = async (rHash: string, destination: string): Promise<string> => {
// TODO: as of lnd 0.12.0, this won't work as PaymentAddr will be required and the only way to specify it will be with the pay req
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should update this one too asap. Maybe using keysend could be an alternative? Will LND 0.12 also break keysends?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about keysends, but since this code isn't being used right now it's not urgent. This whole method could be a todo stub for the time being since we're not using sanity swaps which is what this is for.

request.setPaymentRequest(payReq);
request.setMaxParts(LndClient.MAX_PARTS);
} else {
// TODO: as of lnd 0.12.0, this won't work as PaymentAddr will be required and the only way to specify it will be with the pay req
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know how I feel about keeping this code. Maybe in the first release after LND v0.12.0 we should remove it?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's basically there for backwards compatibility with older lnd and older xud versions, it's ignored by other peers using any version of xud that includes the commits from this PR. I figure removing it after we've confirmed that xud works with lnd 0.12+ would be fine, in my one quick test using 0.12 for the simtests caused them to break so there's some work/investigation to be done there.

@sangaman sangaman closed this Dec 24, 2020
@sangaman sangaman reopened this Dec 24, 2020
@sangaman
Copy link
Collaborator Author

* got failed MPP swap for LTC/BTC Pair

Do you possibly have the lnd logs from this swap? If not I'll try to recreate myself and see what's happening. The swap very nearly exceeds the maker's outbound ltc capacity, but not quite, so I don't see why it wouldn't work. Let me know if there's a docker image from this branch that you've already built & pushed.

@kilrau
Copy link
Contributor

kilrau commented Dec 28, 2020

Do you possibly have the lnd logs from this swap?

https://user-images.githubusercontent.com/29906866/103022899-e61f0600-455d-11eb-8bc9-25b898d17757.png

Let me know if there's a docker image from this branch that you've already built & pushed.

Could you kindly provide this to speed things up? @raladev

@raladev
Copy link
Contributor

raladev commented Dec 28, 2020

* got failed MPP swap for LTC/BTC Pair

Do you possibly have the lnd logs from this swap? If not I'll try to recreate myself and see what's happening. The swap very nearly exceeds the maker's outbound ltc capacity, but not quite, so I don't see why it wouldn't work. Let me know if there's a docker image from this branch that you've already built & pushed.

bash xud.sh -b mp. It uses 5 days old xud from this branch (that i used in testing)

If u want more fresh xud:
1.https://github.com/ExchangeUnion/xud-docker/actions?query=workflow%3A%22Manual+Build%22
2. Run Workflow combobox
3. Choose mp branch
4. print xud into text field

@sangaman
Copy link
Collaborator Author

Do you possibly have the lnd logs from this swap?

https://user-images.githubusercontent.com/29906866/103022899-e61f0600-455d-11eb-8bc9-25b898d17757.png

Those are only the xud logs right? It's the lnd logs I'm interestd in.

bash xud.sh -b mp. It uses 5 days old xud from this branch (that i used in testing)

Thanks I'll try this soon

@kilrau
Copy link
Contributor

kilrau commented Jan 5, 2021

Also needs a rebase

@sangaman sangaman closed this Jan 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lightning Lightning network & lnd integration
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable LND MPP Tradinglimits account for overall inbound/outbound balance
4 participants