Subscribe to RSS Subscribe to Comments

Gensokyo.org

Gallery News

As most of you have probably noticed, the doujin archives have been inaccessible for about 12 hours now. Some of you may also have noticed that the site’s overall speed has been a good deal better in those 12 hours. These two facts are definitely related, which is what brings us to this topic.

The doujin archives here at gensokyo.org are definitely among the most popular parts of the site; currently, about 80% of the “business” we do here is devoted to providing these doujins. This is becoming a problem, though, as the high demand for them is making the rest of the site suffer rather badly.

This brings us to a decision. There are a few options that we can take at this point; the first would be to try to locate a different hosting service that can handle the load a bit better. I personally don’t think this is terribly likely, however, as the issues we’re experiencing here would likely just carry over to a new site as well, and most of the terms of service I’ve been looking over seem to make it possible for the hosting provider to drastically increase the cost of hosting if this happens.

The second option is to pay for a second hosting plan, keep everything but the gallery where it currently is, and simply host the gallery out of a different provider by itself. It bears the downside of doubling the cost for the site, and again many hosting providers seem to be setting rules against this – specifically banning the use of their services as an online file archive, which is what our gallery primarily is at this point.

The third option, and the one that I currently lean towards myself, is to set up a paid account with an online file hosting service like mediafire and use that to host  the files. The advantage to this is that it’s slightly cheaper than paid hosting with another provider, and with a paid account there would be no banner ads attached to the downloads (which I would prefer to avoid, as I’d like to avoid even the appearance of trying to profit off the original work of the doujin artists we’ve scanlated). The downside is that many file serving services have file quotas based on countries, and the files themselves would have to be individually uploaded and linked, which will take a great deal of work to set up (although probably not that much more than the original gallery software here required).

So with these suggestions in mind, what do you think? If you’d like hard numbers to do your own research, the gallery currently does about 750GB in transfer bandwidth per month; this tends to vary a bit, but has been steadily increasing since the time the site originally opened in September of 2007. Bear in mind that for separate hosting services (of any kind) we’ll need at least this much bandwidth capacity, preferably with a good amount of room to grow.

Leave your comments and suggestions below, and let us know what you think!

Comments

  1. hatefulcunt
    July 31st, 2009 | 9:35 pm

    If the archive with its relatively small files got 850GB of traffic per month there’s no fear of there not being enough interested people to keep a torrent going.

    Accounts, payments etc. are dicky and inconvenient and a pain to implement and I am confident that nobody who has any say is even considering them as options!

    Ideally you would get both the gallery back and a torrent – I’d imagine mass downloaders are responsible for much of the traffic, and a torrent isn’t actually less convenient for us (you need a program for downloading the files anyways, might as well be a torrent client), so just having that as an option should make a dent in the site’s bandwidth usage.

  2. Demonwor
    July 31st, 2009 | 11:17 pm

    Hey I have been going to another site that has your entire gallery (i think) And maybe you should talk with them?

    I’m not sure If I should be doing this but here http://www.freewebs.com/aortic/touhou.htm

  3. August 1st, 2009 | 4:56 am

    @Demonwor
    We originally came from there. That’s why you see every doujin up to the point before we moved here over there.

  4. Anonymous
    August 1st, 2009 | 9:00 am

    What is your budget for a hosting plan? If you could give me some numbers, I will try to find a web host that will accommodate your bandwidth and (I’m assuming) high CPU usage.

  5. anon
    August 1st, 2009 | 8:50 pm

    Mediafire

  6. Yurka-Kun
    August 1st, 2009 | 11:27 pm

    Well we could do a torrent of the ENTIRE archive, and find a way to make little spawnlets of the individual ones for those who dont do torrents, but if we do torrents, you could set up a tracker, enable LPD (Local Peer Discovery) DHT (no clue) and that 3rd bit, with a self hosted tracker maybe? cause on the connection i have alot of the trackers wount work at all, leaving me with no seeds unless DHT or LDP is up. Not only will it allow for easier hosting options, but a self fueled archive download at the same time, other thought was also MF, but please, PLEASE avoid Megafuck and Rapidshit (Megaupload/megaporn and Rapidshare)

  7. NaGeL
    August 2nd, 2009 | 3:49 am

    Torrent is the best becouse its free and its got the least load of all of tht methods.

  8. August 2nd, 2009 | 4:46 am

    If you don’t mind me asking, about how big filesize is the archives? I’d be willing to both seed a torrent off of the sm.org VPS and act as a mirror for the actual image files. I’d be willing to help you guys setup a mirror system so you can distribute the load safely across multiple hosts so instead of trying to find a single host that will handle the entire load. You can have those of us with bandwidth and space to spare to host a mirror like back when Danbooru was having problems.

    I sure as hell have space and bandwidth to burn given the relative lightness of the sm.org forums and the dropped activity on fatechan.net (used to pull 700gb/month now I’m lucky to get 50 :[)

  9. August 2nd, 2009 | 4:48 am

    Not to double post, but I wanted to add I’d hate to see such an important project as the doujin archives go down due to being TOO popular. It’s in the whole Touhou community’s interest to try to work together to keep this project afloat in some manner.

  10. August 2nd, 2009 | 8:53 am

    The entire archive is currently 554 files at 6.6 GB, and we’re getting 700+ GB worth of bandwidth from that. I just hope the admin reads this so that something can be done.

  11. August 2nd, 2009 | 12:40 pm

    You can probably add/do the torrent option then add a “Requesting Seed” function in the site and hope for some kind soul to seed. The problem would be the wait for the seed if it would ever come.

  12. August 3rd, 2009 | 4:29 am

    Only 6.6gbs? I can easily host that and seed it then, and the bandwidth/month is well within my limits. 700gb/month even if there’s only 1-2 mirrors is a drastically reduced load for each host.

  13. Makka
    August 3rd, 2009 | 9:01 am

    What about torrent updating on new releases?
    Also, downloading more than 20 files from mediafire in a row is kinda frustrating, no way to wget it all in few clicks.
    What about… establishing DC++ network?

  14. August 3rd, 2009 | 9:31 am

    DC++ network would be pointless because at that point you might as well just use IRC bots, and it wouldn’t help the people who couldn’t use torrents anyways. Torrenting would be best for batch downloads of the entire or parts of the archive (like say all from C72 or a particular artist), while still having the gallery for perusing it and reading directly.

    A mirroring setup would be the best way to go, and the most straightforward for the enduser as it’d be more or less transparent on their end while making it very easy to spread the load around.

  15. Makka
    August 3rd, 2009 | 9:53 am

    And, no more wget.exe -c http://gallery.gensokyo.org/ ?
    Bawww~

  16. August 3rd, 2009 | 10:04 am

    Problem is it’s very very very bandwidth intensive and unfriendly. Most if not all imageboards block recursive or automated download scripts because it is an enormous load on the server. A better solution would be to either provide torrents or a direct link to an already compressed package for each doujin. I don’t know the gallery’s backend code but it’d be trivial to offer both at the very least, a torrent for batches and/or a direct link to a download safe package.

    A torrent allows the bandwidth burden to be shared very easily while also making it easy to download as well. The only downside is that some people are behind firewalls or ISPs that ban or severely impede torrenting so having a direct-download fallback would be a second option.

  17. Onsokumaru
    August 3rd, 2009 | 6:16 pm

    “This is becoming a problem, though, as the high demand for them is making the rest of the site suffer rather badly.”

    Suffer badly? How badly? I, for one, don’t mind if the site itself is slow or unresponsive, as long as it loads eventually. Maybe there is a different way to optimize the site?

    Anyway, if it has to be done, I vote for a big torrent for the old releases and mediafire for the new ones. I will seed as much as I can, plus I already have most of the files ready for initial seeding.

    Good luck with this predicament! May Suwako help us all…

  18. Anonymous
    August 3rd, 2009 | 6:40 pm

    Yukari-sama,

    At first glance I would agree with Onsokumaru above. Have a different torrent for every month (rolled up into one big torrent per year), and release the newer stuff on something like mediafire.

    The problem is I bet the majority of bandwidth usage is the newer stuff rather than the older stuff, and torrents aren’t suited for changing data. Ideally there would be a torrent-like system that allows the data to be appended to on a regular basis. Torrenting for files that are usually at most 20 MB isn’t really worth the hassle, though.

    Is it possible to throw money at the problem? Find a host that won’t shut you down for what effectively is an archive repository and distribute the cost to the users through voluntary donations? I personally have unused income I would donate to such a cause if possible.

    I know the idea of profiting from the work of other authors isn’t desirable, so set up an account that cannot be used for anything other than hosting costs. Or give the responsibility of hosting to someone else. That way you don’t have to worry about this kind of grey-area intellectual property unpleasantness.

    If this is a possibility, let me know. You have my e-mail address and I will be checking the comments.

  19. August 4th, 2009 | 5:43 am

    I think our best option is going to be to simply host off torrents for the main archive, with new releases hosted on-site (and possibly mirrored, thanks TSO) for about six months. About that often we can fold each set of new releases into the primary download torrent, which should move the bulk of the bandwidth usage away from the webserver (i.e., people going for large chunks of the past doujins) while keeping things convenient for the dedicated fans to grab as new releases emerge.

    With that in mind, I will probably keep the existing gallery offline while I get the files currently on-site sorted out and prepped for our seeders (as well as looking into setting up a tracker for the gallery site itself). I’ll unlock the gallery section itself for the next update (poke me for that, Nietz/Ruka) but keep the back archives offline. If you need to grab older stuff feel free to hit the pestermom or older Gaku archives, but patience is probably best if you can wait – better not to overload -those- servers either, when we’ll have the torrent available before long.

  20. Nimmy
    August 4th, 2009 | 1:22 pm

    I think mirrors are a good way to go. Have like 3 or 4 different off-site mirrors for every release, which would distribute the load (and maybe even keep older releases available for those that want them). If you need a good model, see OverClocked ReMix (four different mirrors, plus torrents). I personally can’t use torrents (my Linksys cable gateway won’t allow speeds above 20kb/s), and file hosting sites are something I prefer to avoid.

    Whatever happens though, good luck with everything. :)

  21. Alexander
    August 5th, 2009 | 2:05 pm

    Only a few cents before I go to bed.

    BitTorrent: I’m on a laptop, on a 256/64kbps connection (read: monstrously slow) and sharing that with other people now and again. That makes me a pretty useless contributor. There’s people behind sucky routers or ISPs who can’t use them too. But I do think they’re the best option.

    Third-party hosting: I am totally against shifting it off to another site. ToS evils, link rot and so on. Speaking of which, touhou.wikia.com might need a lot of links fixed!

    Whatever happens, best to all.

  22. Strifer
    August 6th, 2009 | 4:35 am

    I think some people would love to get a batch torrent of everything up till now, if only for the convenience of having an organized catalogue of all scanlated Touhou doujins.

    I also think that encouraging people to seed the archive torrent around the time of a new gaku update would greatly increase the proliferation of all the material. Throwing some announcement over at 4chan’s /jp/ board would also help a great deal.

  23. Onsokumaru
    August 6th, 2009 | 11:09 am

    I must add that it’s easy to keep torrents up to date. Here’s some proof:
    http://www.mininova.org/tor/1494173

    Also, this would be a great opportunity to rename and fix some filenames like:
    Aki_Eda_and_Zun vs. Aki_Eda_&_Zun
    Maybe even add, in [] brackets, on what Touhou game is the doujin’s plot focused.

  24. etb
    August 7th, 2009 | 2:26 pm
  25. Anon
    August 10th, 2009 | 12:43 am

    I’m very, very new, but I figure it won’t hurt to throw in my two cents:

    Torrents are wonderful, for the people that can use them. A lot of people can’t, though, so combining torrents with something else, such as IRC bots as have been suggested, might be a good idea. It seems to me that maybe this is a case where you might not want to put all your eggs in one basket.

  26. Suomynona
    September 9th, 2009 | 12:47 am

    I would go with Yukari’s suggestion about updating bi-annually and secondary hosts for new releases in the interim periods.

    The torrents should either have version numbers (a la the Touhou Lossless collection) or dates so we can make sure we have the current version.

    The problem with updating torrents is that not everyone gets word of it, and new versions will not be seed-able by old (if you look at the Touhou Lossless torrents, you can see people are still trying to leech the old versions), so make sure to announce updates in as many places as possible.

    In terms of organization, I think the comic names in the torrent should at least be “[Author name] (pub date year/month/day) comic name”, so that they will list in an orderly way (by author then publish date). I would also suggest the update torrents listing the comics that have been added since the prior one (since they’re not always the newest by publish date but ones that people might have decided to work on after it was published years ago).

Leave a reply

Based on Fluidity Banner graphic by MARCH
Questions/comments/concerns/bugs can be reported to