Author: @Web3_Mario
Summary: Recently, I've been looking for new project directions, and encountered a technology stack I hadn't dealt with before during product design, so I did some research and organized my learning insights to share with everyone. In general, zkTLS is a new technology that combines zero-knowledge proofs (ZKP) and TLS (Transport Layer Security Protocol), mainly used in the Web3 space to verify the authenticity of off-chain HTTPS data it provides without trusting third parties, where authenticity includes three aspects: the data source indeed comes from a certain HTTPS resource, the returned data has not been tampered with, and the timeliness of the data can be guaranteed. Through this cryptographic implementation mechanism, on-chain smart contracts gain the ability to access off-chain Web2 HTTPS resources reliably, breaking data silos.
What is the TLS protocol
To deeply understand the value of zkTLS technology, it is necessary to briefly review the TLS protocol. First, TLS (Transport Layer Security Protocol) is used to provide encryption, authentication, and data integrity in network communication, ensuring the secure transmission of data between clients (e.g., browsers) and servers (e.g., websites). For those not in network development, you may notice that some domain names have https as a prefix, while others have http. When accessing the latter, mainstream browsers often warn of insecurity. The former may encounter messages like 'Your connection is not private' or HTTPS certificate errors. The reason for these messages lies in the availability of the TLS protocol.
Specifically, the so-called HTTPS protocol is based on the HTTP protocol, utilizing the TLS protocol to ensure the privacy and integrity of information transmission, and making the authenticity of the server verifiable. We know that the HTTP protocol is a plaintext transmission network protocol and cannot validate the authenticity of the server side, leading to several security issues:
1. The information you transmit with the server may be intercepted by third parties, leading to privacy leaks;
2. You cannot verify the authenticity of the server side, that is, whether your request has been hijacked by other malicious nodes and returned malicious information;
3. You cannot verify the completeness of the returned information, i.e., whether data loss could have occurred due to network issues;
The TLS protocol was designed to solve these issues. To explain, some may know the SSL protocol; in fact, the TLS protocol is developed based on SSL version 3.1, but due to some commercial-related issues, it was renamed, yet essentially it is a continuation. Therefore, sometimes in certain contexts, the two terms can be interchangeable.
The main idea of the TLS protocol to solve the above problems is:
1. Encrypted communication: use symmetric encryption (AES, ChaCha20) to protect data and prevent eavesdropping.
2. Identity authentication: verifying the server's identity through digital certificates issued by third parties to designated entities (e.g., X.509 certificates) to prevent man-in-the-middle attacks (MITM).
3. Data integrity: use HMAC (Hash-based Message Authentication Code) or AEAD (Authenticated Encryption) to ensure data has not been tampered with.
Let’s briefly explain the technical details of the HTTPS protocol based on the TLS protocol in the data exchange process. The whole process is divided into two phases: first is the handshake phase (Handshake), where the client and server negotiate security parameters and establish an encrypted session. Second is the data transmission phase, where the session key is used for encrypted communication. The specific process is divided into four steps:
1. The client sends ClientHello:
The client (e.g., browser) sends a ClientHello message to the server, content includes:
Supported TLS versions (e.g., TLS 1.3)
Supported encryption algorithms (Cipher Suites, such as AES-GCM, ChaCha20)
Random number (Client Random) (used for key generation)
Key exchange parameters (e.g., ECDHE public key)
SNI (Server Name Indication) (optional, for supporting multiple domain HTTPS)
The purpose is to let the server know the client's encryption capabilities and prepare security parameters.
2. Server sends ServerHello:
Server responds with ServerHello message, content includes:
Selected encryption algorithms
Server random number (Server Random)
Server's certificate (X.509 certificate)
Server's key exchange parameters (e.g., ECDHE public key)
Finished message (used to confirm handshake completion)
The purpose is to let the client know the identity of the server and confirm security parameters.
3. The client verifies the server:
The client performs the following operations:
Verify server certificate: ensure the certificate is issued by a trusted CA (Certificate Authority), and check whether the certificate has expired or been revoked;
Calculate shared key: use the ECDHE public keys of both the client and server to compute the session key, which is used for subsequent symmetric encryption (e.g., AES-GCM).
Send Finished message: prove the integrity of the handshake data and prevent man-in-the-middle attacks (MITM).
The purpose is to ensure the server is trustworthy and generate a session key.
4. Start encrypted communication:
The client and server now use the negotiated session key for encrypted communication.
Using symmetric encryption (such as AES-GCM, ChaCha20) to encrypt data, improving speed and security.
Data integrity protection: use AEAD (such as AES-GCM) to prevent tampering.
So after these four operations, the issues with the HTTP protocol can be effectively resolved. However, this foundational technology widely used in the Web2 network has caused troubles for Web3 application development, especially when on-chain smart contracts hope to access certain off-chain data. Due to data availability issues, the on-chain virtual machine does not open the capability for external data calls to ensure the traceability of all data, thus ensuring the security of the consensus mechanism.
However, after a series of iterations, developers found that DApps still have a demand for off-chain data, leading to the emergence of various Oracle projects, such as Chainlink and Pyth. They act as a relay bridge between on-chain data and off-chain data to break this data silo phenomenon. At the same time, to ensure the availability of relay data, these Oracles generally achieve this via the PoS consensus mechanism, making the cost of malicious behavior by relay nodes higher than the gain, ensuring they do not provide incorrect information on-chain due to economic incentives. For example, if we want to access the weighted price of BTC on centralized exchanges like Binance and Coinbase within a smart contract, we rely on these Oracles to aggregate data accessed off-chain and transmit it to the on-chain smart contract for storage.
What problems does zkTLS solve
However, people found that this data acquisition scheme based on Oracle has two problems:
1. Cost is too high: we know that to ensure the data transmitted by Oracle to the chain is real and has not been tampered with, it needs to be guaranteed by the PoS consensus mechanism; however, the security of the PoS consensus mechanism is based on the amount of staked funds, which incurs maintenance costs. Additionally, there is often a large amount of redundant data interaction in the PoS consensus mechanism, as the data set needs to be repeatedly transmitted, calculated, and summarized in the network to achieve consensus, which also raises data usage costs. Therefore, Oracle projects usually only maintain some of the most mainstream data for customer acquisition, such as the price of BTC and other mainstream assets. For specialized needs, payment is required. This hinders application innovation, especially for long-tail and customized demands.
2. Efficiency is too low: usually, the consensus of the PoS mechanism takes some time, which causes the latency of on-chain data, which is unfavorable for some high-frequency access scenarios, as there is a significant delay between the data obtained on-chain and the real off-chain data.
To address the above issues, zkTLS technology has emerged, its main idea is to introduce ZKP zero-knowledge proof algorithms, allowing on-chain smart contracts to act as a third party to directly verify that the data provided by a certain node indeed comes from accessing a certain HTTPS resource and has not been tampered with, thus avoiding the high costs of using traditional Oracle due to consensus algorithms.
Some may ask why not directly provide the ability to call Web2 APIs in the on-chain VM environment. The answer is no, because the reason for maintaining a closed data environment on-chain is to ensure the traceability of all data, that is, during the consensus process, all nodes have a unified evaluation logic for the accuracy of a certain data or execution result, or a kind of objective verification logic. This ensures that in a completely trustless environment, most honest nodes can rely on their redundant data to directly judge the authenticity of the results. However, due to Web2 data, it's challenging to construct such a unified evaluation logic because different nodes may obtain different results when accessing Web2 HTTPS resources due to certain network delays, complicating consensus, especially for high-frequency data domains. Additionally, another key issue is that the security of the HTTPS protocol relies on the random number (Client Random) generated by the client (used for key generation) and the key exchange parameters to negotiate encryption keys with the server. However, we know that the on-chain environment is public and transparent; if smart contracts maintain random numbers and key exchange parameters, critical data will be exposed, compromising data privacy.
zkTLS adopts another approach, which is to replace the high costs of making data usable by traditional Oracle based on consensus mechanisms with cryptographic protections. Similar to the optimization of ZK-Rollup over OP-Rollup in L2. Specifically, it introduces ZKP zero-knowledge proofs and computes proofs based on the resources obtained from certain HTTPS requests, related CA certificate verification information, temporal proofs, and data integrity proofs based on HMAC or AEAD, while maintaining necessary verification information and algorithms on-chain, allowing smart contracts to verify the authenticity, timeliness, and reliability of the data source without exposing critical information. The specific algorithm details are not discussed here; interested parties can explore further.
The biggest benefit of this technical solution is that it reduces the cost of achieving the usability of Web2 HTTPS resources. This has stimulated many new demands, especially in reducing the on-chain price acquisition of long-tail assets, utilizing authoritative websites in the Web2 world for on-chain KYC, thus optimizing the technical architecture design of DID and Web3 Game, etc. Of course, we can see that zkTLS also poses a challenge to existing Web3 enterprises, especially to current mainstream Oracle projects. Therefore, to cope with this impact, industry giants like Chainlink and Pyth are actively following research in this direction, attempting to maintain dominance during the technological iteration process, and will also give rise to new business models, such as transitioning from time-based charging to usage-based charging or Compute as a service, etc. Of course, the difficulty here, like most ZK projects, still lies in how to reduce computing costs to make it commercially viable.
In summary, when designing products, you can also pay attention to the development dynamics of zkTLS and integrate this technology stack in appropriate areas, which may lead to new directions in business innovation and technical architecture.