Blog
 / 
Announcements
Concrete ML

Concrete ML v1.8 : Towards Decentralized Private LLAMA Fine-Tuning

January 14, 2025
  -  
Andrei Stoian

Concrete ML v1.8 marks a major step in enabling privacy-preserving fine-tuning for Large Language Models (LLMs). This release improves the speed and usability of LLM hybrid fine-tuning with an optimized FHE backend specific to LLMs and a new Low Rank Approximation API. 

Additionally, Concrete ML v1.8 now supports Python 3.12, ensuring compatibility with the latest tools and frameworks.

A better API to fine-tune LLMs on encrypted data

Concrete ML provides fine-tuning functionalities for LLMs as a client-server protocol: 

  • The server performs encrypted (FHE) inference on the linear layers of the model.
  • The client performs gradient descent on the fine-tuning weights. 

As a first step, developers would compile a LLM to work with FHE in this manner. Before deploying the model, they can check on their own machine that the compiled model can produce good results. To streamline this workflow, Concrete ML v1.8 introduced  a fine-tuning API inspired by HuggingFace PEFT LoraTrainer, making this process more efficient.

from concrete.ml.torch.lora import LoraTrainer

lora_trainer = LoraTrainer(
    model=peft_model,
    optimizer=optimizer,
    loss_fn=causal_lm_loss,
    lr_scheduler=lr_scheduler,
    training_args=training_args_dict,
    n_layers_to_skip_for_backprop=3,
)
lora_trainer.compile(inputset, n_bits=16)

lora_trainer.train(dataloader, num_epochs=EPOCHS, fhe="execute", device=device)

As the snippet shows, Concrete ML requires a single additional call to [.c-inline-code]compile[.c-inline-code], compared to the HuggingFace LoraTrainer. 

Furthermore, this new version allows developers to efficiently use the GPU to evaluate the compiled model on their own machine, significantly speeding up the development process. Check out this notebook to learn how to use the new API on LLAMA.

Faster FHE LLM fine-tuning

Concrete ML v1.8 adds an optimized FHE backend for the LLM fine-tuning use case. Leveraging the GPU, this backend speeds up computation significantly. The low-level operation that is implemented by the backend is the encrypted matrix - clear matrix multiplication, with the encrypted matrix representing private user data and the clear matrix containing the model’s original weights. 

The key highlight of the backend is the efficient compression – Both input and output are compressed to sizes only around 4 times larger than non-encrypted data.

With the current implementation:

  • Fine-tuning a LLAMA 8B model on 100,000 tokens takes around 70 hours.
  • The estimated cost is around $500 using a decentralized network of 100 consumer-grade GPUs.

This represents a significant step forward, and further optimizations in future releases are expected to reduce cost and latency by a factor of 4.

In summary, Concrete ML v1.8 makes it easier and faster to fine-tune LLMs securely on encrypted data, bringing us one step closer to scalable and decentralized AI solutions. Stay tuned for more updates in the future!

Additional links

Read more related posts

No items found.
Libraries
Concrete ↗ Concrete ML ↗FHEVM ↗TFHE-rs ↗
Products & Services
Privacy-preserving Machine learningConfidential blockchainThreshold key management SYSTEM
Developers
BlogDOCUMENTATION ↗‍GITHUB ↗ FHE resources ↗ Research papers ↗Bounty Program ↗FHE STATE OS
Company
Aboutintroduction to fheEventsMediaCareers ↗Legal
Contact
Talk to an expertCONTACT USXDiscordTelegramALL community channels
Privacy is necessary for an open society in the electronic age. Privacy is not secrecy. A private matter is something one doesn't want the whole world to know, but a secret matter is something one doesn't want anybody to know. Privacy is the power to selectively reveal oneself to the world.If two parties have some sort of dealings, then each has a memory of their interaction. Each party can speak about their own memory of this; how could anyone prevent it? One could pass laws against it, but the freedom of speech, even more than privacy, is fundamental to an open society; we seek not to restrict any speech at all. If many parties speak together in the same forum, each can speak to all the others and aggregate together knowledge about individuals and other parties. The power of electronic communications has enabled such group speech, and it will not go away merely because we might want it to.Since we desire privacy, we must ensure that each party to a transaction have knowledge only of that which is directly necessary for that transaction. Since any information can be spoken of, we must ensure that we reveal as little as possible. In most cases personal identity is not salient. When I purchase a magazine at a store and hand cash to the clerk, there is no need to know who I am. When I ask my electronic mail provider to send and receive messages, my provider need not know to whom I am speaking or what I am saying or what others are saying to me; my provider only need know how to get the message there and how much I owe them in fees. When my identity is revealed by the underlying mechanism of the transaction, I have no privacy. I cannot here selectively reveal myself; I must always reveal myself.Therefore, privacy in an open society requires anonymous transaction systems. Until now, cash has been the primary such system. An anonymous transaction system is not a secret transaction system. An anonymous system empowers individuals to reveal their identity when desired and only when desired; this is the essence of privacy.Privacy in an open society also requires cryptography. If I say something, I want it heard only by those for whom I intend it. If the content of my speech is available to the world, I have no privacy. To encrypt is to indicate the desire for privacy, and to encrypt with weak cryptography is to indicate not too much desire for privacy. Furthermore, to reveal one's identity with assurance when the default is anonymity requires the cryptographic signature.We cannot expect governments, corporations, or other large, faceless organizations to grant us privacy out of their beneficence. It is to their advantage to speak of us, and we should expect that they will speak. To try to prevent their speech is to fight against the realities of information. Information does not just want to be free, it longs to be free. Information expands to fill the available storage space. Information is Rumor's younger, stronger cousin; Information is fleeter of foot, has more eyes, knows more, and understands less than Rumor.We must defend our own privacy if we expect to have any. We must come together and create systems which allow anonymous transactions to take place. People have been defending their own privacy for centuries with whispers, darkness, envelopes, closed doors, secret handshakes, and couriers. The technologies of the past did not allow for strong privacy, but electronic technologies do.We the Cypherpunks are dedicated to building anonymous systems. We are defending our privacy with cryptography, with anonymous mail forwarding systems, with digital signatures, and with electronic money.Cypherpunks write code. We know that someone has to write software to defend privacy, and since we can't get privacy unless we all do, we're going to write it. We publish our code so that our fellow Cypherpunks may practice and play with it. Our code is free for all to use, worldwide. We don't much care if you don't approve of the software we write. We know that software can't be destroyed and that a widely dispersed system can't be shut down.Cypherpunks deplore regulations on cryptography, for encryption is fundamentally a private act. The act of encryption, in fact, removes information from the public realm. Even laws against cryptography reach only so far as a nation's border and the arm of its violence. Cryptography will ineluctably spread over the whole globe, and with it the anonymous transactions systems that it makes possible.For privacy to be widespread it must be part of a social contract. People must come and together deploy these systems for the common good. Privacy only extends so far as the cooperation of one's fellows in society. We the Cypherpunks seek your questions and your concerns and hope we may engage you so that we do not deceive ourselves. We will not, however, be moved out of our course because some may disagree with our goals.The Cypherpunks are actively engaged in making the networks safer for privacy. Let us proceed together apace.Onward. By Eric Hughes. 9 March 1993.