We have proposed a solution for this based on Intel SGX TEEs:
w3f/General-Grants-Program#234

• our solution targets on-premise Intel HW (although cloud-SGX would be possible)
• Our grant proposal doesn't (yet) include modifying the substrate client stated in this issue
• VRF could be a low hanging fruit, but isn't part of our proposal

Remarks to your OP:

• Concerning your suggested API I'd be careful with the add key functionality. I would very much encourage people to let the TEE generate keys, because provisioning keys into a TEE opens an attack vector.
• Moreover, as our concept document suggests, double signing protection is not enough to protect against slashing if you assume full compromise of the validator host. The remote signer should also be capable of block execution for serious slashing protection (therefore, HSM's don't seem like a viable solution. They just provide the "wrong" kind of security)
• you mention that Ledger devices could do double signing protection. Please state that TEEs can do this too (even more flexibly).

We're about to do a new VRF, likely called VRedJubJub, that'll we'll need to support as well, and of course BLS signatures, but they do not add as much complexity here, but of course legder devices cannot produce SNARKs and maybe cannot do BLS signatures.

Just for your information: Zondax is receiving a grant from us to work on a flexible TrustZone-based HSM stack

Changing this in substrate will be very involved, as it introduces a completely different pattern of what the keystore is and how it works. They way it works right now is, that the keystore is a single entity in the system (either in memory or saved on disk), holding different types of keys for different tasks. When a component needs to sign something it asks the keystore for the appropriate keys and uses them to sign the data. Meaning this is a direct, non-blocking API and in doubt the keys holds all information for signing directly in memory–though discouraged, you can keep the key around and reuse it.

This however, proposes a completely different approach how signing works. Rather than the keystore holding the keys, you'd have to submit something you'd like to have signed to it and wait for that to return. Making it an async and indirect API. While not impossible, a range of crates depend on the keystore directly and a range of others imply this pattern (e.g. GRANDPA). Switching these is a pretty large task, touching a lot of code, many of which are sync right now and would become async as a result, with –probably– a big tail of things to have to change in responds to that ;) .

We'd prefer doing this by features, not adding some new Signer trait to every substrate crate, right? We've no roadmap for async fns in traits of course, but this holds even if async fns in traits worked, right?

I don't think that it will be that involved on the Substrate side of signing. It is right that we need some changes here and there. However, aura, grandpa and babe are already async. The trait can just return a Future as result and we wait for the signing. By default with no remote signing the api would be blocking and return directly the signed data.

Offchain signing (imonline) shouldn't also be that hard, we need to call into the host anyway and use block_on to wait for the future, like we do it for http requests.

As everything uses the Keystore behind a trait already, it should really be not that hard to integrate.

I donno if https://github.com/iqlusioninc/armistice is relevant, but maybe good to track if you'd working on this stuff

Changing this in substrate will be very involved, as it introduces a completely different pattern of what the keystore is and how it works. They way it works right now is, that the keystore is a single entity in the system (either in memory or saved on disk), holding different types of keys for different tasks. When a component needs to sign something it asks the keystore for the appropriate keys and uses them to sign the data. Meaning this is a direct, non-blocking API and in doubt the keys holds all information for signing directly in memory–though discouraged, you can keep the key around and reuse it.

This however, proposes a completely different approach how signing works. Rather than the keystore holding the keys, you'd have to submit something you'd like to have signed to it and wait for that to return. Making it an async and indirect API. While not impossible, a range of crates depend on the keystore directly and a range of others imply this pattern (e.g. GRANDPA). Switching these is a pretty large task, touching a lot of code, many of which are sync right now and would become async as a result, with –probably– a big tail of things to have to change in responds to that ;) .

Some implementations might actually be synchronous, such as those based on an on-chip TEE.

I think keystore and signer should be two different independent entities. Actually the concept of a software-based keystore may not always be required.. Substrate should ideally deal with a signer only. This signer may later rely on a keystore or not.

My recommendation is to aim for an asynchronous design to cope with latency issues. Even in the case of fast TEEs, it can affect performance if signing operations require context switches, syscalls, etc.

With respect to the work we did at Zondax in Tendermint, yes we used the HSM in Ledger devices (deserializing votes, checking with a monotonic counter, etc.). Latency in these devices is in the order of tens of milliseconds so an asynchronous approach was very important regardless of running in-process or remotely.

We are now working on a completely new design for Kusama/Polkadot/Substrate with a very much hardened datacenter-quality external device, running in a TEE plus in some models we even have access to an integrated HSM. While running a "lean" node would be possible, it means adding a bigger attack surface that we strongly would like to avoid.

Anyway, I am not sure if this discussion is still active.. though having seen the changes here https://github.com/paritytech/substrate/pull/4925/files. I think a good and quick step forward would be to:
1- Decouple the keystore implementation using two traits (i.e. signer, and keystore)
2- Move the current keystore implementation to another crate

This way interested parties can provide clean alternative implementations.

3- Ideally make sign_with async.. however, I understand that it may require substantial work and you prefer to avoid it for now.

IMO, once the signer/keystore have been fully decoupled and made async.. third-party implementations can define their own API, comm protocol, in-process vs remote approach, etc. I think this is the most flexible approach.

There is still one more complex but important issue. At the moment, signers operate on blobs, so they cannot really know what it is being signed. In some cases, signers may even received hashes of the actual content. This severely limits how smart a signer can be.. meaning, it is not possible to track and design adequate double signing protection schemes.

I would need to dig more into the current substrate implementation, but I wonder if there are a few convenient places that could be extended to provide more information at the moment of signing or this is at the moment scattered all over the code.

Otherwise, I can already see that, at least from my project perspective, the keystore is actually not the point where we need to plug-in but just before GRANDPA/BABE/etc decide to sign and still have an structured object.

@brenzi @jleni this seems like a perfect use-case for a formally-verified microkernel, such as seL4. The microkernel could provide software-based isolation between untrusted components, such as the network stack, and trusted components, such as the signer implementation.

One major caveat is that the main framework that I know of for using seL4, CAmkES, only supports systems where all resources are statically allocated. Ideally, the trusted code should not use dynamic memory allocation, but I am not sure if this is practical.

@demimarie-parity Very interesting! But wouldn't this require self-hosted signer HW? Even if cloud services would offer SeL4 VPS, why would you trust them? They still have access to all memory. Am I missing something?

@demimarie-parity Very interesting! But wouldn't this require self-hosted signer HW? Even if cloud services would offer SeL4 VPS, why would you trust them? They still have access to all memory. Am I missing something?

@brenzi You are not. That is one reason why self-hosted signer hardware should be preferred. The biggest caveat is that not everyone can provide the level of physical security required, and most cannot provide the needed protection against DDoS attacks. Could @kirushik chip in?

Using seL4 has a few caveats:

• It is very much “batteries not included” ― expect to bring your own filesystem and drivers. Fortunately, seL4 is also a hypervisor, and can run Linux/OpenBSD/etc in a VM.
• IOMMU/SMMU support is not verified yet, but could be given adequate funding. Without IOMMU/SMMU support, one must trust all of the drivers, which are a substantial amount of unverified code exposed to untrusted input from the network.
• The verified version only supports a single CPU core. Verifying the multicore version is possible but would require funding.
• seL4 has virtually exclusively been used in the embedded space.

To elaborate: From my perspective, the only advantage of a TEE and/or HSM is protection against attackers with physical access. I believe that equally important, if not more important, is privilege separation a la QubesOS. While Substrate is a substantial attack surface, we can remove much of the rest.

Working with QubesOS and Redox might be a good idea as well.

To elaborate: From my perspective, the only advantage of a TEE and/or HSM is protection against attackers with physical access. I believe that equally important, if not more important, is privilege separation a la QubesOS. While Substrate is a substantial attack surface, we can remove much of the rest.

I disagree with this, TEEs do not have much to do with physical access. Both TEEs and HSMs can provide different (better?) guarantees than QubesOS (basically a Xen hypervisor without ASLR or NX).

I will not write extensively here, to avoid going off-topic, given this issue is mostly about providing an API for teams to provide their preferred security solution. Happy to organize or a Riot channel about this though!

Nevertheless, as there are MANY valid alternatives and approaches, I would strongly suggest to make the architecture as flexible as possible so different solutions can be integrated over time.

To advance this a bit further, especially after merging #4925, here's my line of thinking when it comes to implementing client support for remote signing:

• Introduce a Signer interface that aligns with what we did previously by introducing sign_with into the keystore.

pub trait Signer {
	fn supported_keys(
		&self,
		id: KeyTypeId,
	) -> Result<Vec<CryptoTypePublicPair>, BareCryptoStoreError>;

	fn sign_with(
		&self,
		id: KeyTypeId,
		key: &CryptoTypePublicPair,
		msg: &[u8],
                at_blockhash: &[u8],
	) -> Result<Vec<u8>, BareCryptoStoreError>;
}

• The signer in this case, would be used by sp_io to dispatch sign requests regardless of how the signer is internally implemented.
• The substrate node can be configured to run the signer in 3 modes:

/// Type of the client signer.
#[derive(Clone, Debug)]
pub enum SignerType {
	Local,
	RemoteClient,
	RemoteServer,
}

• Local signing means that we use the internal keystore to perform signing. This can happen in the following manner:

pub struct LocalSigner {
	keystore: Store,
}
impl LocalSigner {
	fn new(keystore: Store) -> LocalSigner {
		LocalSigner {
			keystore,
		}
	}
}
impl Signer for LocalSigner {
	fn sign_with(
		&self,
		id: KeyTypeId,
		key: &CryptoTypePublicPair,
		msg: &[u8],
                _at_blockhash: &[u8],
	) -> Result<Vec<std::primitive::u8>, BareCryptoStoreError> {
		self.keystore.sign_with(id, key, msg)
	}

	fn supported_keys(
		&self,
		id: KeyTypeId,
	) -> Result<Vec<CryptoTypePublicPair>, BareCryptoStoreError> {
		self.keystore.supported_keys(id, vec![])
	}

}

• RemoteClient type of signer is where the substrate node dispatches signing requests towards a specific host/port over a specific endpoint for signing, be it an HTTP(s) call or a gRPC or potentially other protocols.

• RemoteServer, on the other hand, tells the substrate node that it should open a port and listen for "secure" connections where the node can send signing requests over this connection.

• We could also abstract the protocol implementation into it's own trait so that additional protocols can be implemented on top of this where incoming / outgoing payloads can be encoded / decoded.

• To enable double-signing protection to be implemented by the server, it is suggested that the sign_with interface also adds at_blockhash parameter where the signing requests explicitly define the block hash at which signing should happen. This enables the server to query certain blockchain information such as block height or other parameters required.

• The interface is defined to be "sync" here but could use block_on to perform async operations. That is, until async support is implemented in certain parts of the substrate codebase.

I would like to get some feedback on the above to move this forward.

Why do you want to introduce a new trait? The KeyStore trait is exactly meant for this, as abstraction over the key store.

RemoteServer should be an extra application and should not be included into the Substrate node!

You don't need to pass at_blockhash to the sign function. Based on the key type, you can decode the opaque blob that should be signed and this blob already contains all the information you need to prevent double signing.

Why do you want to introduce a new trait? The KeyStore trait is exactly meant for this, as abstraction over the key store.

You're absolutely right. After working on the code for a bit, it is apparent to me that the separation of Signer and Keystore doesn't make sense. I am reverting the work i did by keeping Keystore as-is and going to introduce RemoteKeystore which handles remote key management and signing.

You don't need to pass at_blockhash to the sign function. Based on the key type, you can decode the opaque blob that should be signed and this blob already contains all the information you need to prevent double signing.

Could you expand on this a bit please? how would the key type be relevant to the blob sent for signing?

If you see the KeyType that is used by Babe, you can just decode the blob to the Babe specific structure. The same goes for Grandpa. Every key type makes it possible to identify the encoded blob to decode it.

I've increasingly realized that block seals should probably use the extra arguments to VRFs in https://github.com/w3f/schnorrkel/blob/master/src/vrf.rs not a separate signature, but not worth the effort required to change this since it'd only save 64 bytes per block.

@burdges I would love to see that change be made sooner rather than later, but I am not sure if it is practical right now. We can always make it at the next hard fork.

Hi, when can we have a remote signer for substrate session keys?

I just re-read the description and everything is still very relevant. Let's prioritize this?

We've several major projects that shall further change the session key crypto: beefy, including optimized signing, sassafras, including ring VRFs and ephemeral block signing keys, new session certificates for shashing reform, post-quantum options, and equivocation prevention.

All development is path dependent.. It's possible if complex to implement remote signers for these after they're working, but it's impossible to implement & deploy these once everyone expects a remote signer.

It should also be mentioned, that Zondax has a working external signer that allows the management of session keys inside of an ARMs TrustZone, however it seems Parity has not interest to support this officially yet (see #10423 for details).