libp2p integration

[fjl]

libp2p is a modular framework of peer-to-peer networking components,
implemented in several languages. Specs can be found here.

We want to have a shared transport protocol with libp2p to enable interconnectivity
between the IPFS and Ethereum networks. Transport protocols specified by libp2p
could be used as a replacement for RLPx.

See also libp2p/libp2p#33.

[raulk]

Raúl from the libp2p team here 👋

A bunch of thoughts:

• libp2p features a kademlia DHT implementation with slightly different behaviour. Currently this is used by many projects including IPFS and Polkadot (with their rust-libp2p impl).

  • We're working on other DHT implementations as well, such as Coral.
  • There's a DHT research group in Protocol Labs tackling several aspects such as security, routing, scoring, etc.
  • Some ideas for improvements:
    • Collection of ideas: https://github.com/libp2p/research-dht/issues/6
    • Secure disjoint path lookups: Implement secure disjoint path lookups libp2p/libp2p#44
    • Data structures: Analysis: k-bucket underutilisation and excessive thrashing libp2p/go-libp2p-kad-dht#194
  • By adopting libp2p, we hope the efforts we're investing to improve DHT lookups, security, connectivity, etc. will benefit Ethereum as well.

• RLPx handshake (ECIES) and SecIO share similarities. I collected my thoughts here: What is Secio? libp2p/go-libp2p-secio#7 (comment)

• ENR (Ethereum Name Records, https://eips.ethereum.org/EIPS/eip-778) could be materialised via the Interplanetary Record System. Spec here: https://github.com/libp2p/specs/blob/master/IPRS.md

• Multiplexing can be handled by Yamux. We support ls messages to list protocols (thus playing the role of wire-level HELLO in devp2p), then through a single connection, any number of virtual streams can be opened for each protocol/conversation.

• We currently don't send DISCONNECT messages. Some discussion taking place here: "disconnect" protocol/message libp2p/go-libp2p#238

• For integration with other languages (e.g. python, C++, Java, etc.), the libp2p daemon is under active development these weeks. This is a standalone process which allows local processes to interact with remote peers through libp2p, using simple Unix sockets and shared memory transports (in the future). Implementing bindings in other languages is straightforward. In fact, there's already a Gerbil binding, and other folks have expressed their interest in contributing Java and Python bindings.

There are probably many other relevant topics that don't spring to mind right now, but I will be watching this issue. I'm excited to help the Ethereum community through this journey wherever needed ;-)

P.S. If you need to chat with me, you can find me in the #libp2p Freenode IRC channel (nick: raulk), or in the Ethereum Sharding Gitter ;-)

[vyzo]

cc myself

[raulk]

@fjl What do you have in mind for the interconnectivity between IPFS and Ethereum? Any particular features you're interested in?

[fjl]

Whoa, so many suggestions at once.

RLPx handshake (ECIES) and SecIO share similarities.

RLPx is broken. I want to replace it. I've opened this issue because I want to integrate some of the libp2p transport protocols, in particular TCP+secio and UDT+secio, as replacements for RLPx.

By adopting libp2p, we hope the efforts we're investing to improve DHT lookups, security, connectivity, etc. will benefit Ethereum as well.

I think we have a solid plan for devp2p node discovery and will continue working on the protocol that we have for that.

ENR (Ethereum Name Records) could be materialised via the Interplanetary Record System.

Btw it's Ethereum Node Records. I don't understand 'materialised' in this context. Can you clarify what you mean?

[fjl]

What do you have in mind for the interconnectivity between IPFS and Ethereum? Any particular features you're interested in?

The idea is to be able to cross-connect the systems on the network level. If I understand IPFS correctly there should be a way to address content stored in other systems. I just thought it might be possible to make Ethereum a system like that. You could fetch blocks this way, for example.

[raulk]

Whoa, so many suggestions at once.

Apologies for the flurry. Since I've worked with devp2p before, I had been cooking up some thoughts for a while and I figured I'd share them.

RLPx is broken. I want to replace it. I've opened this issue because I want to integrate some of the libp2p transport protocols, in particular TCP+secio and UDT+secio, as replacements for RLPx.

Cool, good to know.

I think we have a solid plan for devp2p node discovery and will continue working on the protocol that we have for that.

👍

Btw it's Ethereum Node Records. I don't understand 'materialised' in this context. Can you clarify what you mean?

Apologies for the typo! I was thinking Ethereum Node Records would be stored in the DHT somehow, but I don't think that's the intention, so the connection isn't clear to me now.

The idea is to be able to cross-connect the systems on the network level. If I understand IPFS correctly there should be a way to address content stored in other systems. I just thought it might be possible to make Ethereum a system like that. You could fetch blocks this way, for example.

This is a neat idea. IPLD allows traversing different data types stored in IPFS (including ETH blocks). See https://github.com/ipfs/go-ipld-eth/blob/master/plugin/README.md. Currently one needs to import blocks manually into IPFS, so it would be helpful to have ETH clients with IPLD/IPFS capability push blocks onto IPFS real-time, or expose an IPFS/IPLD protocol so that other IPFS clients can pull data from them by advertising themselves in the IPFS DHT.

[fjl]

Apologies for the typo! I was thinking Ethereum Node Records would be stored in the DHT somehow, but I don't think that's the intention, so the connection isn't clear to me now.

Node records can be stored in any DHT because they're just small binary documents. Our discovery DHT stores them, anyway.

[raulk]

Our discovery DHT stores them, anyway.

Where can I learn more about this? Our discovery DHT means the ENR variant of the discovery DHT? From what I know, the v4/v5 discovery protocols didn't store data in the DHT in any way (in the Kademlia PUT_VALUE, GET_VALUE sense).

[debris]

cc @tomaka

[fjl]

Yes, it doesn't store any values. But node records are not values, they're just pointers to nodes with a bunch of metadata attached. The libp2p/ipfs equivalent to node records are multiaddrs, but these are not signed or versioned.

Where can I learn more about this?

There are two efforts to bring ENR support to the discovery DHT:

• Discovery v4 ENR Extension (EIP-868: Node Discovery v4 ENR Extension #44)
• Discovery v5 (Specify Discovery v5 #48)

Both of these are being worked on by various people.

[FrankSzendzielarz]

@raulk Please see the discv5 subfolder for the work in progress.

Hello. Any updates on this effort? Any pointer? Or new link / issue?

Hola @raulk 👋 !!!

[fjl]

Discovery v4 ENR extension is done. We're still working on Discovery v5. Next step for me is finishing discovery v5, then I'll start looking into libp2p and transport stuff again.

[fjl]

Discovery v5 is now pretty stable, we just released version 5.2, but still working on it.
Regarding the transport, I really want to work on a UDP-based transfer protocol next.
Not sure if there is any overlap with libp2p's interests right now.

We also have issue #71, where some people got really excited about building some kind of TCP transport with libp2p, but the discussion seems to be over in that issue.