The Hard Truth about the Cost of Providing Free ROMs and Monthly Updates…
So,people have gotten used to get everything for free from projects like ArrowOS or other custom ROM creators. Not once did they stop to think about what it really takes for that to happen. People seem to believe that it is perfectly normal to pop up in a group chat and demand a download link for the new ROMs available. They barely even say thank you these days…
So that’s why I decided to to write a quick post to share some light on what it really takes to make that happen. How much storage space and how much bandwidth is needed for that magic to happen. Here’s a quick example: Let’s take one type ROM alone, let’s say the Gapps build which is usually around 1.5GB. Most of the users will show up on our webpage and download a ROM for their device. The thing is, the ArrowOS project is not only for their own device.
ArrowOS runs on like, 100 different devices… Now; multiply 1.5 with 100; You’ll end up with 150… That’s GB’s. Only for the Gapps ROMS.
Please note that I have not included THE VANILLA BUILDS,THE EXPERIMENTS,THE UNOFFICIAL builds AND ALL THE TEST BUILDS the developers are working with, the ones the general public never gets to see. The ones that Developers test over and over and over again, tweaking them, perfecting them until they meet the standards for public release. Believe it or not, those take up space as well…
I WAS EXTREMELY CONSERVATIVE … Even in those conditions,in order to produce ONLY one ROM for each device it still takes a lot of storage…
It takes 150 GB of storage EACH MONTH to produce a ROM and another 150 GB to issue an update later on say, at the end of the month… EVEN WHEN YOU DO IT ONCE PER MONTH FOR ONLY ONE TYPE OF BUILD. We’re talking just one type of ROM and a one time update for that ROM here…
So, we ended up with 300GB worth of stuff on our server ALREADY. Now, storing the files IS ONLY ONE PART OF THE EQUATION… You have to be able to DISTRIBUTE them to the users as well…
And here is where the fun begins…Or better said, where the shit hits the fan…
We ALREADY have 300 GB of stuff that needs to be distributed to the users this month… AGAIN, A VERY, VERY, VERY CONSERVATIVE ESTIMATE…
Let’s assume our users are all super nice, and none of them IS A DICKHEAD that uses DOWNLOAD MANAGERS which make multiple requests to server for different parts of the file creating huge traffic and overloading the server’s CPU’s…
Let’s assume none of them use Linux and download accelerators (LIKE AXEL) where you can manually set the number of connections (axel -n 100) which of course hammers the server each second WITH EVEN MORE REQUESTS… 100 per second in the case above.
Lets assume NONE OF THE DOWNLOADS FAIL,and the users don’t just close the download and proceed to start ALL OVER AGAIN…
Let’s assume our users really care for us and for our server and they make a backup of the file they download so they don’t have to request the same file OVER AND OVER AGAIN… Let’s assume all that…
In these perfect conditions, in THIS IMPOSSIBLY PERFECT WORLD, under the best case scenario, UNDER THE MOST CONSERVATIVE scenario where you ONLY HAVE ONE TYPE OF BUILD (No experiments, no unofficial, no vanilla build, no test builds and so on… ), EVEN IN THIS SCENARIO, you are still looking at an expected transfer around 300GB of data monthly.
Now that might not look like a lot, but let’s be honest here, this will NEVER BE THE CASE… ArrowOS PROVIDES MORE THAN ONE TYPE OF BUILD which will take up waaaay more space than 300 GB of storage, and THE REGULAR USER OUT THERE IS AN OUTRIGHT DICKHEAD using all kind of download managers AND DOWNLOAD ACCELERATORS hammering our servers with requests and ripping through our data allowance WITHOUT ANY CONSIDERATION for the poor server, for us, and for our modest budget.
Which brings us to the next point of my explanation… BANDWIDTH… People EASILY overlook this aspect of the opensource projects… BANDWIDTH and HOSTING…
Bandwidth is a term that most of us have heard of but not many understand. Although they always like to pretend they do…
It is basically the amount of data transferred over the internet connection within a specific time. It is EXTREMELY IMPORTANT TO UNDERSTAND that you have to plan in advance for how much bandwidth you are using because it can affect your hosting costs,performance,and more importantly your bank account.
There’s two ways you can go about it :
Metered Bandwidth
With a metered bandwidth plan, you only have available a set amount of bandwidth every month. That’s REALLY NOT IDEAL.
This means that once this allowance is used up,by the dickheads I mentioned above our website or server can no longer be accessed until the next billing cycle begins. That means until WE PAY AGAIN...
Shorter put, in a language everyone can understand : NO MONEY, NO HONEY…
Or, we could go via the more ambitious way, the Holy of the Holiest…
The Magical Unmetered Bandwidth Hosting Plan:
This type of plan allows you to use an unlimited amount of data within the allowed disk space on your server. This is what you want…This is perfect when you need to send timely updates (let’s say, monthly updates) to your users. But, as always, things are not that simple…They never are… Because in real life there is no such a thing as UNLIMITED bandwidth… There’s something called the “fair usage policy” which most of the web hosting providers strongly enforce…
In order to ensure the normal operation of the server and websites hosted there, they do require all customers to comply with their Acceptable Fair Use Policy and utilize their disk space and bandwidth IN A FAIR MANNER.
This “Fair manner” varies from provider to provider. They all have specific conditions and limits set for CPU, Memory Usage, Entry Processes and amount of inodes. In some cases THEY WILL put constraints on your account, if your resource usage is impacting overall server performance.
Such an unfortunate situation might occur very easily when a particular customer (A project like ArrowOS that NEEDS to send out a LOT of data in the form of updates to its users monthly) utilizes some shared hosting account to store and share GIGABYTES of DATA (REMEMBER THOSE NUMBERS WE TALKED ABOUT ABOVE? ?).
When this allowance is exceeded,the provider reserves the right to slow down or suspend your site until another monthly cycle begins (usually at a time when traffic will be lighter).
As you might have begun to realize at this point, running a project like ArrowOS and keeping ROMS available for our users 24/7 is A VERY EXPENSIVE venture.
ArrowOS keeps copies of Android ROMs starting from Android 9 and ending with A13. Those are not only taking storage space up, but believe it or not, they keep SUCKING UP BANDWIDTH everytime when a user or two hundred of them get nostalgic and decide to have another go to an older ROM because: “A11 bess mere bhai”.
And as you can imagine, as the time passes, that archive of older ROMS keeps growing and growing and the TRAFFIC it generates eats a LOT OF BANDWIDTH which of course costs money.
As I said earlier, ArrowOS runs on over 100 devices (last time i counted).
Now quick reminder again :
Assuming that for each of those devices, we upload a GAPPS copy only which weights around 1.5 GB, when you multiply that by 100 , which is the number of the devices we support ,you’re obviously ending up with 150 GB (That’s ONLY FOR ONE BUILD at a specific time).
Even in those conditions,in order to produce one type of ROM and have it available for download, let me emphasise it again, ONLY one ROM for each device it still takes a lot of storage… It takes 150 GB of storage EACH MONTH to produce a ROM and another 150 GB to issue another update later on say, at the end of the month… EVEN WHEN YOU DO IT ONLY ONCE PER MONTH. We’re talking updates here…
So, we ended up with 300GB worth of stuff on our server already… As I said, if to those 300GB worth of ROMs you begin to issue updates AND CREATE THE VANILLA VERSIONS for those ROMs (which most of the devs do, that seemingly little number begins to grow exponentially ). When you factor in the VANILLA versions, now you have an extra 160 GB of your storage gone.
IN TOTAL ,YOU ENDED UP WITH 360 GB gone. That’s for a month worth of ROMS (vanilla and gapps builds along with their once per month updates)
When you do THE MOST MINIMUM updates (once per month ) a whole year of data will be like 4320GB . That’s 4.32 Terabytes…
ONLY For one year providing the MOST MINIMUM updates. IMAGINE THE TRAFFIC that creates. Because let me assure you: NO ONE downloads their ROM only once. For some reason you have users downloading their ROMs even three times (because they can’t be arsed to keep a backup copy).
Now imagine the 4.32 TB of traffic created only by the two types of ROMs… But guess what? ArrowOS has been around for waaaaay longer than that… We have been providing people with updates (sometimes even daily) since the days of Android 9.
So , I am sure you guys can imagine just how much storage space and bandwidth a project like ArrowOS used since those days… Imagine YOU trying to keep all those copies and DISTRIBUTE THEM… Imagine that… And from this moment on, we already begin to enter the professional territory…
There’s gonna have to be some really hard questions asked: All this data stored is only part of the story….
How readily does it have to be accessible? Real Time? Minutes? Hours? Days?
Does that apply to all the data or just the most recent? The past month? The past year?
WHO’S GONNA PAY FOR THIS? What are the penalties incurred if THE TRAFFIC GOES OVER the allocated plan? Who’s gonna pay the costs?
Speaking of costs… The elephant in the room everyone pretends does not exist while acting in the most selfish manner and demanding instant updates, instant fixes and more features like “Unlimited Google Photos storage”… The costs…. Let’s have a look at how much would cost to have a website WITH ONLY 300GB storage… Just for fun…
Shall we?
Right… that does not seem that bad, I hear the entitled snowflakes saying… You know them… those snowflakes that believe they are particularly unique and special and they deserve a lot of handouts a lot of freebies and basically everything for free. Just because… Anyways, that’s what it would cost to be operational just A MONTH…How about one year? two? three, four Sure… Let’s have a look at four years plan
Yeah… shocking, right? Remember, that’s for a four year plan only… And that was with Hostinger which is actually not that bad…
You see the problems the project is facing now? Now I am sure you understand why for example we cannot afford to keep EVERY ROM ever produced available 24/7.
So… Yeah… For those of you that still might think I was exaggerating have a look at some real time data and see how many TB of traffic we are dealing with… 77 TB going and let me tell you, that was light traffic …
We normally go through 90 TB each month … Well, now that you know, you can’t pretend you don’t know anymore…
Maybe knowing all that can help people understand that they can keep the project afloat by donating. You can’t pretend you don’t know what is going on… The choice is yours…
do you really need to host everything on your site? Why not make torrents and encourage the community to seed? Also, can’t you host releases on Github for much cheaper (iodeOS does this)? Other roms use sourceforge or other providers for hosting.
Maybe I’m missing something, but we absolutely do not need direct links, hosting the releases on any service would be just fine, and if it goes down, simply reupload.
And while we’re talking, is Note9 discontinued or only 12.1?
Thank you for the rom and keep up the hard work
Yeah, sure… Because those people who are so self absorbed and so selfish that use ADBLOCKERS even when they come to visit our site to download A FREE GIFT which we provide, are so nice that would be willing to seed and have their computers upload GB’s of data…
Those are the type of people unwilling to give anything back!
They know perfectly well that by using an adblocker we can’t even get the little add revenue we would otherwise might have gotten from the adds…
But they do it anyway…
Do you think those people are actually willing TO SEED and have their computer upload tens of GB’s only so they can help us?
It would actually cost them way more than allowing adds for ten minutes in their browsers and they know that as well…
In reality we would be lucky if we could get a seeder or two maybe for each ROM.
Have you ever tried to download a movie where there’s only one or two seeders who have set the upload rate to the most minimum and the rest are leechers?
Try it, is gonna be an interesting experience.
Reference the github do the following experiment…
Try to monthly upload anything between 300 to 500 GB (our roms and archive collections weight way more than that) to their servers and afterwards try to create a 90 to 100 TB of monthly traffic (which is what we get through monthly) on their site…
Come back and let us know how it goes.
Kind Regards
Mountain Gorilla 🙂
It works for Linux distros the world over. The vast majority of which whose users are simple consumers who will never contribute back. It works.
You my friend are what we call an “arm-chair administrator”. People should not talk about things they have no idea. Let me summarize it for you: It actually doesn’t. It does NOT work…
Debian, Ubuntu, Fedora, Opensuse,and Slackware they all provide direct downloads from their website. There’s a reason for that. The smaller projects might offer torrent links as well but that is the exception not the norma.
These kinds of slightly amusing and frankly a bit ridiculous suggestions are usually made by young people (usually in their 20’s) who don’t understand how the torrents work and what’s involved in administering a project of this size.
You can’t blame them,most of these arm-chair administrators were not even born when this technology first appeared. BitTorrent appeared in 2000 when the internet infrastructure was really really in its infancy.
Websites were not that many and their capabilities were PRIMITIVE compared to today’s standards. My beloved Debian 2.2 (Potato) was just released and everyone was creaming in their pants over how mega advanced that release was, Ubuntu, which most of the people think to when you say linux wasn’t even born yet…
It would be another 4 years until the first version of Ubuntu would appear and because the first versions were so shit, most of us would use Debian of course…
Those were dark times with primitive internet and primitive infrastructure. That’s when BitTorrent was born…
Now… BitTorrent’s original intent was to provide a simple mechanism to alleviate the problem of downloading large amounts of data when there is no established mirroring infrastructure in place, or the mirroring infrastructure is incapable of handling the demand put upon it.
In 2001, this was a serious concern, as it was quite possible for large and popular datasets to cause the meltdown of both servers and network infrastructure.In some cases, this caused noticeable slowdowns and bottlenecks on the entire Internet. If you were a linux user during those times you would know what i am talking about.
BitTorrent’s intention was to come to the rescue by distributing the combined load to every user who was participating. By taking advantage of the aggregate resources available, users were than able to download faster, and in downloading,helped make downloads for others faster by also uploading the content that a user has.
What you proposed, has already been tried by Fedora (in their famous experiment ) and by the most popular linux distributions.
The data (The real one ,not the one the arm-chair administrators imagine) proved something i already said in my previous posts: that only a very few number of
nodes were performing the vast majority of the work in the BitTorrent cloud.
This was because people were leaving the cloud once they have completed their downloads as there was absolutely NO advantage for the user to continue uploading into the BitTorrent cloud after acquiring the full download.
This behaviour leaves the cloud increasingly dependent on the few seeders who have a full copy of the ROM. Which of course, from our point of view it will mean that the users are not fully updated when they should be, that means that the users will experience cumbersome downloads that take days and days like the way it used to be back in year 2000. That means a less than satisfactory experience for our users. That’s not good!!! We have a reputation to maintain. A reputation which we worked extremely hard to achieve. I am sure you begin to realize the complexities of the problem by now.
Now, BitTorrent in its current implementation, Is a massive pain in the ass when it comes to the distribution or mirror administrators to set up. Try it for yourself… Just for fun… Is gonna be an eye opener.
The administrator must put together package sets and create the torrents MONITOR THEM AND UPDATE THEM. Here’s some quick math and some facts (which most of the people ignore because facts are boring and NOT cool. )
ArrowOS runs on over 100 devices. Try to remember that… 🙂 This is both important AND COOL.
Now , Imagine… Vanilla builds , Gapps Builds, Experiment builds, and Unofficial builds. Are you with me so far ? So far we are talking 400 different torrents JUST FOR ONE RELEASE .
If you add updates into the equation, ONLY ONCE PER MONTH which is (extremely ,extremely conservative) NOW YOU’RE TALKING 800 torrents.
Those 800 torrents will have to be recreated every month, or at least every update, every bug fix every little modification ArrowOS introduces. We are talking about a lot of man-hours here (is the amount of work performed by the average worker in one hour).
Our project can simply not afford that much work, as we are an extremely small team working unpaid and only survive because of the donations. There’s only so many hours in the day and so many days in a week and all of us have real jobs commitments which we can not afford to miss or ignore as we all have kids,family and bills to pay.
Most of the developers would barely have two hours per day available to work on the project and they have to make a hard choice between family life (wife kids girlfriends ) and /or working on the ArrowOS project.
I’m gonna be brutally honest here… People who are talking nonsense from the comfort of their chair saying stuff like “It works” without thinking, are not exactly the smartest persons on the planet.
And it easy to see that they have absolute no experience with mirroring / hosting / and admin work.
Now, we are in year 2003… Bittorent is mostly a thing of the past and increasingly associated with illegal downloads. What does that mean? That means everyone is cracking down on the use of the torrents and Bittorrent clients.
In EU ,USA,UK and pretty much everywhere else BitTorrent’s are throttled down, or simply prevented to access the internet (unless they continuously change ports) .
Their users are being served with notices from the internet provider about possible violations of the law. Most of the BitTorrent users are not using Deluge which allows you to force encrypt the traffic both ways and only accept secure connections where the traffic is scrambled and encrypted and rotated across various ports so the internet provider does not recognize it as BitTorrent traffic. They just use their other ,substandard BitTorrent clients and guess what. If they live in the western hemisphere their experience is less than ideal.
Now, you have to understand that the classic distribution mechanisms (rsync HTTP, ftp) are very simple for both distribution and mirror administrators.
Simply put the files in a downloadable location, and the mirrors download the data to their servers. When the time is right, the distribution and the mirrors perform what is commonly known as a “bit flip” (or a changing of the file permissions) to allow normal users to acquire the data. This is quite simple for both parties;
There is no way in the hell a project the size of ArrowOS could be run on torrents alone. The sheer work involved is tremendous.
One day when i will have some time i will write more in the blog about it… Believe it or not this was the super ,mega short explanation. 🙂
All I’m going to say is there are no armchairs here.
that feel when no a13 for lavender
U should create Torrents for every file
Kindly read above.
The attitude of this blog post (and especially the author’s responses in the comments) is awful and ensures I want absolutely nothing to do with this project. Instead of insulting and shooting down everyone who makes helpful suggestions to solve the issues in this blog post, the author should try being more kind and welcoming.
I would also say that the attitude of some people that can’t seem to read the blog properly and rush to make vitriolic comments can be awful.
Most of them seem to have read only small parts of the blog. If you read it carefully you will realize that this post it was meant as some sort of an eye opener to the general public (which can be extremely demanding and unreasonable).
As you are aware most of them seem to believe that everything is free and that there is no hard work and sacrifices involved in maintaining a project. They also like to conveniently turn a blind eye to the fact that there are ACTUAL,REAL LIFE costs involved with maintaining an infrastructure.
If you read my post carefully you would have realized that this is only a general example, as the figures i presented in there are extremely conservative. Web Hosting IS costly depending the size of your archives the plans you chose and some providers can be more expensive than other as you can see from the Hostinger’s example.
Now , once again this was meant to explain the fact, evident to the senses,and to everyone that has a bit of admin experience, that there are costs involved with maintaining a project.
Now, everyone seems to interpret this posts in his own way and whilst we got some really good suggestions indeed from some people (I won’t mention them here, they know who they are), most of the people, the vast majority, seems to be jumping at the chance to make vitriolic comments and sometimes offer advice with a side of snide remarks, making comments about “Skill issues” and ranting from the comfort of their armchair about how they could have hosted the whole project on a better faster cheaper plan, heck even run the whole project for free…
I mean is evident to anyone who stops to look at our website even for a second, (WHICH MOST OF THE PEOPLE COMMENTING DID NOT) that we are not with Hostinger 🙂 however people seem to be focussing on that aspect alone and use it to make mean spirited and outright vitriolic comments.
Once again, this post was meant to shine a light on what it takes to run a project the size of ArrowOS and to highlight the fact that however you look at it, there ARE costs involved.
Anyways, arguing in the comments section with everyone is a loss of time.
Kind Regards
Gorilla
Nobody came here to argue, friend. Even though your post is largely a rant and not informative or representative, you’ll notice most here still started out with suggestions to help.
The post was very informative. I think you’re one of the spoilt brat the author was talking about 🤷
my friend Gorilla! I am simply amazed by your endurance and patience in explaining simple things to an ungrateful client (for the fact that he is ungrateful, he says that he, without objecting to the essence of what was said, still stands his ground, thereby devaluing your work!). I wish you success in your hard work and …. thank you very much for the excellent custom !!
All right, so, it’s a pain. I get it. And I actually was thinking by myself recently – where are all those modders getting the money back when they are providing such a service. I mean, I am software engineer by myself and I know the struggle. I know how much pain it is just to test stuff, since I am actually a software tester and hardware tester (for 3D printers also). And what I am hearing here totally makes sence, Gorilla. I am actually kinda glad about that because this means one think for me – you are not implementing any malicous tools into your ROMs that would financially help you with the project like selling user’s private data. So, I am kinda prepared to support you. But I am lacking some stuff. Like for instance – you are saying that your ROM is functional on 100devices and even more. But here, on the page, in the list of devices it’s maybe 30? I wasn’t counting to be honest. Like for instance, I came here to this page because I found the post about ROM on XDAdev forum for the Xiaomi Pad 5 deivce. But I can’t find that device in the list. It’s confusing for newbie like me to actually understand what to download and how to do the root for my device (in this case the Xiaomi Pad 5 device).
Second thing – “how much is the fish ?” Meaning, I can throw some “dolazz” your direction, but how much is enough? And how to understand your work? Meaning – can I understand it as a “licence for operating system” which means – I can install it on any device? Or is it for specific device? You get my point, right?
I counted. Theres 101 supported device. A good chunk of which is just xiaomi, poco and redmi devices.
Huge respect for your hard work. I would suggest you to make donate option more obvious. Like make “Donate” button next to “Download” button.
I didn’t even know you are accepting donations until I bumped into this article.
Damn ! This really did put things into perspective for me.
.”…Como usted sabe, la mayoría de ellos parecen creer que todo es gratis y que no hay trabajo duro ni sacrificios involucrados en el mantenimiento de un proyecto… ” Absolutamente de acuerdo con esta expresión. Ciertamente debemos reconocer su tiempo, su dedicación a la persona que desarrolla estos proyectos…. No es posible que todo lo veamos gratis , gratis.
But why do I have to seed or allow adds in my browser. i don’t want to be traced by Google. And why can’t you guys provide more builds of android 11? Lots of people need it. Why do I have to make a backup copy when you guys can keep this for us . This is 21st century . If you are a project you should know that your users are always right and have a right to access older builds. Why are you even in this project if you can’t plan properly? You should plan accordingly and make sure builds available for user.
Not sure if you’re being sarcastic but if not YOU are exactly the kind of user this post is talking about. If you’re not satisfied why don’t you go fuck off somewhere else? How about being grateful for the hard work they put in?
I really admire all the work and effort youve put in but can you please not be such an asshole to people trying to make genuine suggestions? They may not understand or know of all the problems but theyre acting in good faith and just trying to help. And dont just assume the people in this blog are adblocking and are butthurt you called them out, thats dumb.
i can understand all time and money spend for making 1 rom, but here I just wana suggest something to take help from YouTube… other youtuber who doesn’t even know how to build rom or even requirements to make rom, telling everyone that they can build rom on termux and doing so called reviews video of rom in which too they just talk rubbish… it would be lot better if also start making yt video explaining how you making these roms for different devices and how someone else can make or giving basic knowledge required to use Linux, etc. many topic are their you can make video on and offcourse once you got 10k subs u can get ads revinue and sponser money too…