Lilypad Network Announcements

📢 Update: Work in progress to launch RP Beta program
👋 gm Resource Providers! 🐸

Work continues to rearchitect the Testnet rewards system including planning for the Resource Provider Beta program. In order to align the rewards system closely with how RPs will operate on Mainnet, the rewards system will require RPs to run jobs in order to receive rewards. More on this soon!

Thank you to all RPs who submitted the Beta program form. We will make an announcement when the list of Beta RPs has been decided. Our team is currently implementing the new rewards system and intend on starting the Beta program as soon as the system has been setup and testing has completed.

Sign up for the Beta program with this form - https://forms.gle/sLDrgUXeV7HpCmSq8 !

📢 Update: Retroactive rewards drop status

👋 gm Resource Providers! 🐸

A retroactive Lilybit rewards drop has been planned and is on schedule to be applied early next week. The retroactive drop will provide Lilybits to RPs that are missing rewards through no fault of their own due to IncentiveNet scaling issues.

Stay tuned for more IncentiveNet updates as we rearchitect the backend system and begin testing the next round of updates!

Our team cannot thank the community enough for the support as IncentiveNet has grown over 2024! We are energized by the growth of the network and are full steam ahead towards Mainnet launch.

Incentivised Testnet Update

gm gm Resource Providers

After multiple fixes for the recent issues with Resource Providers not receiving rewards on the Incentivised Testnet, we’ve decided to temporarily pause the system. This pause will help address scaling challenges in the Lilybit rewards system and make necessary enhancements for Mainnet.

During the IncentiveNet pause, we’ll rework the rewards system to resolve outstanding issues, with no expectation for Resource Providers to maintain uptime while points aren’t issued. We’ve started reworking the backend, and the rewards calculation will align with Mainnet goals, including a job-based mechanism to simulate the future rewards system.

- The Lilybit InventiveNet rewards will pause on Monday Dec/16/2024 and is expected to resume in mid to late January.
- A retroactive drop of Lilybits will be given to RPs missing rewards through no fault of their own up to Dec/16/2024.
- A snapshot will occur after the retro-drop to record Lilybit rewards earned by each RP. RPs will not lose Lilybits during the IncentiveNet pause.
- TestNet (IncentiveNet) will run with a small beta group of up to 20 RPs earning limited rewards during this time.
- The intention of the small beta group is to ensure TestNet can consistently run jobs. We will need RPs who are able to test and can share details about their deployment.
- When the rewards system is resumed, we will announce that Resource Providers can join the public IncentiveNet.
- RPs that do not return to running on the TestNet (IncentiveNet) when the public rewards begins will be subject to slashing.
- Lilybits earned in phases 1-6 will not be included in slashing as thanks to early network supporters.

Applications for the beta program are open now via this form https://lilypad.team/betaprog

Stay tuned for more updates as we continue to work on the re-architecture of the rewards system. Thanks for your continued support and patience as we work towards improving the system!

📢 Update: Rewards system investigation

👋 gm Resource Providers! 🐸

We are aware that nodes across the network are not receiving Lilybit rewards as expected. Our team investigated this issue and believe the root cause have been identified. The Lilybit rewards system continues to face scaling issues, so in response, our team is scaling up the system backend to handle the increased load. This work has been on-going over the last week as we carefully push updates and evaluate the results without causing additional complications.

In addition to implementing a fix, we will complete a retroactive backdrop of Lilyibt rewards for RPs missing rewards through no fault of their own. An announcement will be sent out to the community when the retro drop is completed.

Follow along in more detail at https://github.com/orgs/Lilypad-Tech/discussions/18 and https://discord.com/channels/1212897693450641498/1307591874982182982/1307591874982182982

🚨 Reminder: Lilypad v2.9.0 and older have been deprecated. RPs running v2.9.0 or older will not receive points. Running the latest version of Lilypad is critical to the ability of a RP to run jobs and earn rewards.

Check out our GitHub discussions board to see current work or report an issue: https://github.com/orgs/Lilypad-Tech/discussions/categories/rp-issue

📢 Update: Rewards system investigation

👋 gm Resource Providers! 🐸

Over the last week, our team has pushed backend updates to the Lilybit rewards system and continues to resolve points issues with some Resource Providers (RP).

An important part of the fix includes scaling up the rewards system to handle the large volume of RPs online. We will keep the community updated as this work continues. A huge thanks to the community for supporting the Testnet as it scales!

Follow along in more detail at https://github.com/orgs/Lilypad-Tech/discussions/18 and https://discord.com/channels/1212897693450641498/1307591874982182982/1307591874982182982

🚨 Reminder: Lilypad v2.9.0 and older have been deprecated. RPs running v2.9.0 or older will not receive points. Running the latest version of Lilypad is critical to the ability of a RP to run jobs and earn rewards.

Check out our GitHub discussions board to see current work or report an issue: https://github.com/orgs/Lilypad-Tech/discussions/categories/rp-issue

📢 Update: Rewards system investigation

👋 gm Resource Providers! 🐸

Our team greatly appreciates the patience while edge cases of some RPs not receiving rewards are investigated.

We have been running test cases to identify the root cause and believe this is related to how the rewards system backend handles recent rewards changes. Testing continues and in the meanwhile we are specing out a fix. This has taken some time as the community brought up a few different edge cases that required investigation to get the full picture.

Follow along in more detail at https://github.com/orgs/Lilypad-Tech/discussions/18 and https://discord.com/channels/1212897693450641498/1307591874982182982/1307591874982182982

Note: Our team may be slow to respond towards the end of this week as it is a holiday in some countries where we operate. That said, we are full steam ahead towards a resolution for these rewards edge cases and will further communicate next steps as soon as possible.

Check out our GitHub discussions board to see current work or report an issue: https://github.com/orgs/Lilypad-Tech/discussions/categories/rp-issue

📢 Update: Lilybits rewards emissions

👋 gm gm Resource Providers!

The 20,000 Lilybit daily limit per RP has been implemented and will take effect for Lilybit rewards moving forward. Read more about this update in this announcement. https://discord.com/channels/1212897693450641498/1256179769356189707/1308230244167192728

🚨 Mandatory Update: Lilypad v2.9.0 Required by Tomorrow!

gm Resource Providers! 🐸

Reminder that the previous Lilypad version (2.8.0) will be deprecated tomorrow. RPs must update to v2.9.0 - Einstein to remain active in the network!

Upon every release, RPs will have 2 weeks to update to the latest Lilypad version, mentioned in the v2.9.0 release - https://discord.com/channels/1212897693450641498/1256179769356189707. Please note: The next Lilypad release will be tomorrow and RPs will have a 2 week grace period to update and then v2.9.0 will be deprecated.

Resource providers that are running older versions will no longer remain active on the network, have PoWs tracked or earn rewards. Make sure your node is updated to v2.9.0 before tomorrow to avoid interruptions!

Update Linux RP:

1️⃣ Stop the RP (skip if offline):

sudo systemctl stop bacalhau
sudo systemctl stop lilypad-resource-provider

Leave the local IPFS node running.

2️⃣ Remove the old Lilypad executable carefully:

sudo rm -rf /usr/local/bin/lilypad

3️⃣ Install the latest version

https://docs.lilypad.tech/lilypad/hardware-providers/run-a-node/linux#install-lilypad

4️⃣ Restart RP

Once configured, start the RP:

sudo systemctl start bacalhau
sudo systemctl start lilypad-resource-provider

Update Docker RP:

Use the docker-compose file to effortlessly setup the RP. https://docs.lilypad.tech/lilypad/hardware-providers/run-a-node/docker#docker-compose

Got questions? Open up a ticket in our GitHub discussions using the rp-issues category: https://github.com/orgs/Lilypad-Tech/discussions/categories/rp-issue 🚀

📢 Update: Implemented initial fix for Rewards System

Our team has implemented a fix that will address the large number of daily Lilybit emissions. Changes take effect today and will reflect in the total daily Lilybits sent out going forward.

* Only one wallet (RP) will be eligible to earn rewards per GPU.
* For RPs running multiple wallets on one GPU, the wallet with the most points will be matched in our system with the GPU and will earn points. The other wallets will not earn points going forward (unless they are used on a different GPU).
* Slashing will continue to be turned off temporarily (as a grace period) until we are certain the rewards system is working as expected and to allow time for RPs to adjust. Slashing will likely be turned on mid/end of next week.
* Once turned back on, slashing will affect any address that does not have a unique GPU associated with it.

We'll communicate next steps on rewards emissions concerns once enough data has been collected on the results of this work. https://github.com/orgs/Lilypad-Tech/discussions/14#discussioncomment-11261561

To report additional RP issues, open a new github discussion by filling out this template or check out any existing discussions for your issue. https://github.com/orgs/Lilypad-Tech/discussions/new?category=rp-issue

For troubleshooting help, open a ticket in the i-need-help Lilypad Discord channel. https://discord.com/channels/1212897693450641498/1230231823674642513

📢 Update: Slashing temporarily paused

👋 Hi Lilypad Resource Providers!

The Lilybit rewards calculation was updated and based on community feedback a bug was discovered for RPs running multiple GPUs on one server. RPs running v2.9.0 on a server with a single GPU should not be affected and are asked to continue running v2.9.0.

While our team decides the next steps to address this, slashing has been turned off temporarily. We will notifiy the community when slashing is turned back on.

IMPORTANT NOTE: With the new rewards calculation (using the daily rewards multiplier and GPU info VRAM + Tensor cores + Cuda cores), RPs should expect to receive a different number of Lilybits per day when compared to rewards earned using the previous hashrate based calculation.

🚨 Action Required - New Release - Einstein (v2.9.0)

"Imagination is more important than knowledge." - Albert Einstein

gm Resource Providers! 🐸

Lilypad v2.9.0 - Einstein was released requiring RPs to update! https://github.com/Lilypad-Tech/lilypad/releases/tag/v2.9.0

Changes include updates to the Lilybit Rewards calculation, an update window, & more. The Lilybit calculation will change this week to using GPU card info instead of hashrate. Slashing will be turned on this week.


🚨 Update Window

With new releases, RPs will have two weeks to update before the previous version is deprecated. Failure to update in this period will prohibit a RP from earning Lilybits.


v2.9.0:

* To fairly reward GPUs optimized for AI jobs, Lilybits will be provided based on GPU card info (VRAM, Tensor cores, Cuda cores) in addition to the PoW uptime multiplier. The calculation will no longer use hashrate, but RPs will still be required to run PoWs. https://blog.lilypadnetwork.org/update-to-the-lilybit-rewards-calculation
* RPs will report their Lilypad version to the network
* This release includes a docker-compose file for RPs to easily get setup and running! https://docs.lilypad.tech/lilypad/hardware-providers/run-a-node/docker


Update Linux RP:

1️⃣ Stop the RP (skip if offline):

sudo systemctl stop bacalhau
sudo systemctl stop lilypad-resource-provider

Leave the local IPFS node running.

2️⃣ Remove the old Lilypad executable carefully:

sudo rm -rf /usr/local/bin/lilypad

3️⃣ Install the latest version

https://docs.lilypad.tech/lilypad/hardware-providers/run-a-node/linux#install-lilypad

4️⃣ Restart RP

Once configured, start the RP:

sudo systemctl start bacalhau
sudo systemctl start lilypad-resource-provider


Update Docker RP

Use the docker-compose file to effortlessly setup the RP. https://docs.lilypad.tech/lilypad/hardware-providers/run-a-node/docker#docker-compose

Got questions? Hop into our Discord #i-need-help channel. 🚀


https://lilypad.team/discord

📢 Update: Rewards yesterday

👋 Hi Lilypad Resource Providers!

Yesterday, the Lilybit rewards mechanism experienced a technical scaling issue resulting in all RPs not receiving points for 2024-11-05. Our team has investigated and continues to scale up the rewards infrastructure to prevent this from occurring again.

Lilybit rewards are set to go out at 00:10 UTC as usual today. RPs should also expect to receive the missing rewards for 2024-11-05 (yesterday). Regardless, this affected RPs evenly. We will be actively monitoring the system to ensure recent updates address the scaling issues.

We greatly appreciate the community's patience as IncentiveNet scales. Our team continues to balance between network scalability, cost, time, important updates, and Mainnet development. The Lilypad pond continues to grow and we cannot be more excited for the future of the network!

🪷 Lilypad ambassador rewards for October 🪷

Hi Lilypad Advocates!

The next round of monthly Lilybit Ambassador rewards were sent out today. Thank you for the hard work over the last month, your contributions are invaluable to help the network grow!

Lilybit_ rewards can be found at https://oss.lilypad.tech/.

The Lilypad Ambassador program is designed to highlight and support outgoing and passionate community members who help others, solve problems, and communicate effectively!

Learn more about becoming a Lilypad Ambassador https://lilypadnetwork.notion.site/Lilypad-Ambassadors-f11f73e91f684fa192fc2fab7985fe0d.

📢 Update: Rewards today

👋 Hi Lilypad Resource Providers!

Our team has investigated and resolved issues with an Arbitrum RPC used for the Lilypad Testnet core infrastructure (IncentiveNet). RPC errors experienced today were generally caused by problems around scaling high volumes of transactions and were mostly out of our team's control.

As a result, all RPs across the network were affected with 0 Lilybit rewards earned today (2024-10-29 in https://rp-points.lilypad.tech). With this in mind, our team has implemented a fix to the rewards system, but will not provide retoactive Lilybits. RPs were not slashed today.

We apologize for this situation and thank our community for the incredible support as Testnet grows!

@everyone 🚨

Our APAC AMA is live NOW!

Bring your questions along and chat with Steve and Taosheng.

Come join us on X/Twitter
https://x.com/Lilypad_Tech/status/1846370890009006098

or join us on YouTube
https://www.youtube.com/live/1KCO1bMoNzA

🪷 Initial Lilypad ambassador rewards are here! 🪷

gm friends! @everyone

We're happy to announce that the first round of Lilypad ambassador program rewards have been distributed! 🎉 Covering contributions from August 27th to September 30th, our advocates are being recognized for their incredible work with Lilybits.

From developing tooling that improves the resource provider experience, to helping with debugging, to offering positive support within our community, every effort has made a difference. 🌱 These contributions are driving Lilypad forward, and we're so grateful for everyone’s dedication.

Find the list on the "Ambassador Rewards" tab at https://oss.lilypad.tech/ !

This is just the beginning, with more exciting updates on the horizon for our community contributors. If you've participated, check out the dashboard to see your rewards—early contributors and ambassadors even got a special bonus. Thank you for being an integral part of the journey this early on! 🐸

If you're interested in joining the Lilypad ambassador program, find out more information here: https://lilypadnetwork.notion.site/Lilypad-Ambassadors-f11f73e91f684fa192fc2fab7985fe0d

Our Pondside Chat is live NOW!

Come join DeveloperSteve and Edatweets pondside 🐸

🔗 https://www.youtube.com/live/K90A7uDFny0

🚨 Lilybit Rewards Update

Hello Resource Provider!

Lilybit Rewards went out as usual at 00:10 UTC today.

Our team has noticed a notable decrease in the amount of Lilybit rewards earned by RPs network wide (today 2024-10-05). A new phase started ~4 days ago, so rewards sent out should have decreased by only 10% from last week. The decrease has been more than 10%. This has affected all RPs equally, thus will not require a retroactive backfill of rewards.

We are investigating and will let the community know when this has been fully addressed.