Archive for the ‘BanLink’ Category
Welcome to the Future
I'm alive! And I've returned. And I've even updated my look so I don't look like I was born in the early days of SL! New hair! Wow, never thought I'd be able to wear new hair. So here I am. And no, I've not done a lot. I don't know if I will do a lot. Just taking it easy and seeing how this new SL is working out.
I will, however, talk a little about Ban Link. I've been out of SL for about a year. During that time I've poked at Ban Link a time or two, but not a whole lot. Travis has been handling and managing everything. I even got to where I was ready to move away from it. I didn't, however. And that was more a situational decision than anything. We were going to phase me out, and then Travis, of all people, disappeared.
So what is the future of Ban Link, you ask? Is there a future? Well, indeed there is. Hopefully not too little, too late. I've already moved Ban Link onto new hardware, and it's been performing brilliantly. The database has been tuned up and all responses are just thrilling. We went from processing 5,000 Ban Link messages in a minute, to over 10,000. And not only that, our average time per message went from around 1-2 seconds to less than 1/4 of a second. It's really great news and has really inspired more work.
And yes, there's been more work…
The web side of Ban Link has been getting a much needed overhaul. For those that have been around a while, you'll remember we moved from Ban Link Web 1.0 to Ban Link Web 2.0 a long while back, and that was a much needed interface improvement. However, the initial plan for that was to provide much more in the way of improvements than came about. Finally, Ban Link Web 3.0 is near completion. And it involves an entirely new framework. So this time around we won't see a whole lot in the way of interface changes, though there will be some significant updates in how the data is presented, but what we will have is a much more robust system and hopefully along with the server improvements, a system that will operate in a much more usable fashion.
That being said, I'm already going to make some aggressive changes to Ban Link's user base. This is partially due to the severe backload of registered but inactive groups in the system. As we know, Second Life changes weekly, daily, hourly, and is one of the reasons we made sure a group that wanted to come online was one with significant traffic and not likely to be here one day and gone the next. If you're an active user, you have nothing to fear. If you've requested a group for your place or have an account that is inactive. it will be disappearing. But anyone is very welcome to register again and get back into our queue.
Another change? Better tracking of history. And this is in nearly every respect. The original website grew from a very small project into what it is today. And we basically chased a moving target to get to this point. The new improvements will help us in administration and hopefully you as well in determine what's working and what's not. Who's doing what, and even better tracking of which groups are more or less trustworthy than others.
It's an exciting time, and a lot of things are happening. I hope the new changes will be welcome and thanks for your continued support.
BanLink, the Quest Continues
More BanLink updates are in the works. This time ban aging. It wasn't exactly a needed feature two years ago when all of this started, but we've got some really old bans out there and they don't do much more than clutter our lives. So we're implementing some ban aging. It can be set per group so not everyone has to let a ban age away, but the idea is to allow groups to set how long they wish for a ban to remain active.  After that period of time, the ban will no longer be in effect. For 90% of the cases out there, this should be great. We realize the vast majority of bans from griefing come from accounts that are either now canceled, deleted, or just plain unused. And for those that may have acted up in the past, this gives them the ability to return after having "grown up" a little or simply no longer feel punished due to a misunderstanding. Yeah, it happens.
On a similar note, it irks me to no end when I see an account holder on BanLink abuse the system. I realize this sort of this happens. But it still is very bothersome. This system has been made to contain griefing as best as possible. It's also to be used to keep various troublemakers from causing trouble around various places on the grid that have a like-minded way of operating.
Every so often, however, I'm informed of a group that's using the shared ban list for personal grudges. That is wrong and I wish I had a way to make that stop. We've tried to put in checks and balances to inform other groups as to who is submitting "valid" bans versus those submitting…well, we'll just call them "childish" bans by providing numbers on bans and disputes. It's not the best system, but it's something.
What's the problem, you ask? Well, if a group has a personal grudge against someone, then they can easily create a private ban. That keeps the individual off of their land, but doesn't notify anyone trusting them that the person has been banned. And that's how things should be. You're allowed to keep anyone off of your land that you wish. But barring someone from entering the land of others when they've really done nothing wrong in a broad sense is abuse to the system, and one of the few complaints I still hear. It's a valid complaint.
Travis and I have a policy of non-involvement. It's precisely how BanLink should be. This is not a singular and global ban list. There are not two people (us) arbitrating disputes and allowing or overturning bans. The system is distributed and each group has full control of who is banned from their land. No one group has more "power" or standing than another. We're all equals and we're going to keep it that way. This means that we can't go in and overturn a ban that's been issued, even if it's an abusive kind. We can simply encourage the banned individual to file a dispute and hope that the other groups that trust the group creating bad bans realize the pattern and distrust them. It's an effective solution, but not a quick or perfect one.
So here I am with a plea. We've kept the system free. We've kept the power to the people. We even try to keep things running as smoothly as possible at all times. So *PLEASE* don't abuse the system. You signed up with BanLink, hopefully, to join in and help each other keep abuse from those that are trying to disrupt SL down to a minimum. Don't be an abuser yourself. Ok?
Thank you!
BanLink Goes Linden
Just kidding!
BanLink was recently mentioned on the Linden blog. Unfortunately we were inaccurately listed as a reputation/rating system. BanLink is *not* a reputation/rating system. BanLink is a way for groups to share ban lists and manage their own individual lists independently. We give the power for each parcel to have complete control over who is banned from their land and who is not, while also providing them a means to trust other groups as to who should and should not be banned.
What are some of its features?
- Manage their own ban list, while accepting trusted bans from other groups
- Decide which other groups to trust
- If a person is banned from their land via a trust, they can be unbanned into their own land without affecting the originating or any other bans.
- If a person is banned from their land and they are the originator of the ban, then they anyone that trusts them will have that same individual banned from their land as well. However, if the originator unbans someone, the person will also be unbanned from all trusted land parcels unless explicitly banned from another parcel.
- BanLink is decentralized. Parcel owners have complete control over who is banned and who isn't, regardless of trusts.
- BanLink includes a dispute system which will allow individuals who have been banned to respond to the reason for the ban. The dispute is visible to not only the originating ban owner, but by anyone that trusts the originating group. This allows for trusted groups to make their own determinations as to whether they should allow the ban to exist on their own land.
- BanLink is a free system.
What does BanLink not do?
- We are not a rating system.
- We do not provide public information on individuals who have been banned. We believe in privacy and are not in the business of building a reputation.
- We are not a central government. All bans and unbans are managed wholly by the groups in the system. We do not intervene with any form of favoritism based on groups or individuals. If there is a dispute, it must be brought up with the groups and individuals who have initiated the actions. We are not a judge and jury in any matter.
- We are not a system to be used by hot-headed groups who will ban people out of a personal grudge. If a group has a tendency to do this, then chances are that the group will not be trusted by anyone else and therefore have no real impact to others using the system.
- We are not tied to Linden Lab, nor are we being taken over by Linden Lab, as has been rumored.
This is a project written by Travis Lambert and Mera Pixel (hey that's me!), to help alleviate potential griefing situations by quickly sharing ban information amongst groups of like-minded individuals. The system works pretty well and is being tweaked and improved on a near daily basis. We also include improvement ideas not only from our group subscribers but those that have been banned as well. If anyone has an idea, hopefully well thought-out, that you think would benefit the system, please bring it up to one of us. We're looking to make this useful and fair to everyone involved.
The Trick with Database Optimizations
There's always a catch, right? Of course there is. One thing to take into account with database optimizations is that you don't optimize the database over the system it's running on. In other words, don't let it take up so much memory that the system is swapping memory to and from the hard drive swap space constantly.
I ran into this problem a few weeks ago with BanLink. The system was having all kinds of difficulties and then noticed that it'd gobbled up so much memory that it was spending half the time swapping the database cache from the disk cache.
Another catch, at least in my case, but I've read of a few others, is that MySQL will not *always* restart after you've changed the database settings. In other words, it shuts down fine, but just refuses to start. The solution I've found to this is to blank out the configuration, start it up, apply an updated configuration, and restart. It doesn't *always* work, and I don't even really understand why it should work, but it's about all I have to go on.
I experienced this a few weeks ago, only I couldn't get my new settings to take. No matter what I tried, the database refused to start with any optimized settings. So I just let it start up unoptimized. That worked. The stats on the site weren't even all that bad, so I let it run for a while.
That was short-lived, however. A good test of your optimizations are to run it without them for a while. Ok, that's a really poor test. Averaging around 10 seconds to respond was not a pretty thing. But I finally managed to get some of my new settings pushed into the database. We're back to sub-second responses, and we're not eating up all the memory of the system to do so.
The point? A good DBA pays attention to their database and doesn't let it win the battle…and it will fight you.
BanLink Update
Ok, time for another of my wonderful BanLink updates. I haven't done this in a while, but I should. Even if all of you hate it, it still is nice to keep track of what's going on. It'll be fun to look back on at least.
The database, thank goodness, has been running pretty smoothly lately. I did have to push in a few additional memory tweaks last Wednesday to make sure it's caching as best it can, but so far the load on the server has been minimal. This is a very good sign considering we've more than tripled the amount of traffic that the site gets in the last month.
The latest work has been completely web related. I'm excited and looking forward to the BanLink 2.0 web release. And yes, it really is coming. I've spent a lot of time rebuilding the core of the system and am now putting it within a pretty interface. Using all the modern technologies available, like AJAX routines, the site is becoming much easier to navigate and hopefully will provide for a much nicer user experience overall.
When I built the original site, it was done in a fairly rushed fashion just to get the web and SL to interoperate. The more interactions we needed, the more was added to the site. This is what accounts for the lack of cohesiveness in a few of the screens. It's also why the system seems so slow at times. Communicating back into Second Life via RPC is slow, especially if the sim that the RPC box is housed in is running a little sluggish. It'll get there, but it may take a few seconds. Add to that the sites that have 20 boxes to communicate with, and you find yourself sitting and waiting and *hoping* things are working. The new site will have a much more logical layout, meaning less need to hop between screens to do similar things. It's also going to help with communications as they will now work in parallel. And you'll get a status message to keep you up to date on what's happening. Isn't that nice? Yes, it is.
There's good stuff to come. Thanks to everyone who has supported the BanLink system. It's nice that we're hearing more and more success stories as of late. It helps to keep us going when we know the system is providing use to folks out there.
A sneak peak, you say?
Continuing the Database Optimization Tips
It's been pointed out to me that I should remind everyone that these techniques can really only be applied when you have complete control over your MySQL instance. Many users in a hosted environment don't have this control. However, most users won't need to worry about this sort of tuning. It is important, however, when you have a heavily used system, such as BanLink, where you're getting constant transactions 24×7.
So this is part two in our MySQL Database Tuning series. Last time we looked at the number of opened tables versus the number of tables cached. Next we're going to look at the index buffer. The trick is that you want to give as much memory as you can (go figure) to the index buffer without allocating all of the memory in your system.
The rule of thumb is to set it to a quarter to half of the available memory in your system, provided MySQL is the primary use of the system. How can you tell when you've got enough memory and things are working great, well we again return to the extended-status that we talked about last time. There are two status variables that we'll need to look at: key_read_requests and key_reads. You'll want to divide key_reads by key_read_requests. You wan that resulting number to be less than 0.01. If it's not, bump up the key_buffer value in your my.cnf file.
A companion to these values are key_writes and key_write_requests. As above, divide key_writes by key_write_requests. This time you want the result to be less than 0.1. Again, this is affected by the key_buffer setting in your my.cnf file. Once you have it tweaked to where both your reads and writes are optimal, your database will be nicely humming along.
BanLink & Databases
Time for my weekly BanLink update, I suppose. After last week's changes, looks like I've got things on track finally. We're processing a little over 92,000 commands from the in-world objects per day and the average response time is less than 1 second and the server load average for the day is .24. In other words, things have finally come together. I'm all for sharing knowledge, so I should use this time to talk about what was actually done. I won't post everything all at once here, mostly because I could go on forever at a time, but consider this part one of a multi-part posting.
First off, this is a MySQL database. MySQL is fast, lightweight, and free. Plus it ties in to most languages pretty easily. I'd fiddled with tuning MySQL a little in the past, but in general it's never been really necessary outside of commercial applications. Well this site made it necessary, obviously. So what do you do? First step, get some stats out of MySQL by running this command:
mysqladmin extended-status
This will output a whole bunch of status information for your to peruse. Remember that this status information is also a snapshot of what's happening right when you ran it. So it's best to run this during peak performance times so you can get a "worst case" view of what's going on. Your MySQL database will have a configuration file, typically called my.cnf and living in /etc. You will want to make your changes within that file.
Now, what should you look at first? Well one choice is to look at open_tables. This status will tell you how many tables are currently open. There is a corresponding value called opened_tables. That one, obviously, will tell you how many tables have been opened. Now check in your my.cnf file for the setting, table_cache. If your table_cache setting matches your open_tables value, and the opened_tables value is much higher, it's a good idea to bump up your table_cache value. This will allow MySQL to cache more tables. Yay for caching. If, however, you see that your table_cache setting is higher than your open_tables value, then you know you can decrease the number of tables you need to cache. No point in allowing MySQL to cache too much and take away memory for no reason.
Thus ends part one. Hopefully I'll be able to explain things more clearly as I get more practice at this. In the meantime, happy optimizing.
BanLink Update
It's time for a BanLink update. We've been working pretty hard on this system the past few weeks. Travis is gearing up for a new in-world release, and I've been continuing with the back-end optimizations. The plan is to get the database into working in a more streamlined fashion, and then work on updating the website. I know those of you that use the website do not think it's ideal, and it's not. The website was more or less thrown together at the start of this whole project and bits and pieces were put together as it grew. Now that we know more of how the site can and needs to be used, we can update it and make it more functional. We have already received a list of enhancement requests from you folks that have been using the system, much of those had been on our own list of enhancements since the beginning. Now, finally, they will come. Thanks for waiting. 
Back to the latest bit of work, though.
Continue reading
I’m happy and sad for you
Well, happy and sad for me, actually.
The server tweaks that I've done have really made a difference, but I'm a bit confused by what I'm seeing, I have to admit.
You see, the improvements really did work on the server as a whole. The load average stays pretty low most of the time. The average in a 24 hour period is 1.2, but during light times it hangs out at 0.5. This I like. What's rather surprising, however, is that the execution time of the controller script is at around 1 second. I really expected it to drop to sub-second execution. In fact, with it running for so long, I really expected the server load to be higher. This is what's bothering me. How is it that the app is running without a lot of improvement, but the server load has improved dramatically? My suspicion before was that the DB was the bottleneck in the whole process and once that was improved, the app would return immediately. I guess it really is time to fix that program and see where the hangup is.
Speaking of evil. Tuning Databases is right up there. Sure, you get to run through and check table reads, index caching, memory allocation, etc. It's like a big puzzle that you get to analyze and put together. You definitely have to be in the right mood for it or it can just be a world of suck. But the bigger problem is when you make changes and then the database refuses to restart. Sure it exits just fine, but the important part is starting back up. Twice I've had the database refuse to start after making some tweaks. And it doesn't really complain about any of the changes. To make matters worse, untweaking doesn't help. So far what I've found is that I have to replace its configuration file with one that's empty, start the server, replace it with a tweaked config file, then restart. Things then seem to go just dandy. I don't make the rules, so don't blame me if that sounds silly.
Time for Tech Talk
Since I went ahead and posted something, I guess the flood gates have been opened. Well let's get into something substantial, shall we? Yes, we shall.
The past few weeks I've been reobsessed with BanLink. It's been running fairly well for about four months now. During that time period, both Travis and I have taken breaks from BanLink and even SL in general. We'd been working on it for a while before its release, so we deserved it, dangit. Thankfully, it runs pretty well on its own. I've popped additional enhancements and tweaks into the web site from time to time, and Travis has done the same to the in-world boxes. With each release, the system gets a little better. Most of this "better" is transparent to the end-user, but system-wise things are improving.
None of this is to say I haven't had a large to-do list for the web side of things. There is a lot on there, much of which is fairly easy to do, but I needed the inspiration to do it. The real fun stuff, however, sort of kicked in in the past month, when I decided to pay closer attention to the servers running this shebang. The servers, I noticed, have been running with a fairly high load average. Jumping up past 20 on occasion. Now these are dual CPU machines, so 20 is a tad on the excessive side (we'd prefer less than 2, for those unsure of what the heck a load average is).
It was time to dive into the servers themselves and take a closer look. It seems that there has been a whole lot of email traffic being queued up, and bounced, and retried, and so forth, which wasn't pleasing. After some digging, I discovered that boxtrapper had been enabled on a couple of user accounts. Boxtrapper is a spam blocker that runs on the server. It's the kind that automatically responds to your email message when you send to someone, requiring you to confirm that you're not a spammer. Once you respond to it's automated email, it'll allow the original one to go through. In theory, it works great. In practice, it sucks up disk space and CPU time. I'd had it disabled for quite some time, but apparently some had found it prior to that disabling and had it running on their accounts. Once I was able to disable boxtrapper completely, the servers breathed a sigh of relief.
But it didn't end there. Oh, no. The servers, my friends, were still being hit hard. The load average was hanging out at around 1.8 for much of the day. Under light load, like in the morning when less seem to be in SL, the load average would be around 0.7 – 0.8. I can handle a load average that's less than one. That would be wonderful, in fact. But it just wasn't consistent enough. It was time to take a look at the stats on BanLink to see what was going on in there. I've had ideas on how to optimize things, but hadn't implemented anything major in quite some time. After watching the process list for a bit, it was clear what was going on. The BanLink controller app would run just fine for a while, then about six would be kicked off at once at which point the database would go into overload. I knew the database would eventually be an issue since it really didn't take long to surpass one million records, but in general the database tables are pretty lean. I didn't expect the DB to become quite the bottleneck that it had become. It was time for some performance tuning.
After some poking around, I'd realized that I hadn't really tuned the database at all since it's install on this server. Well that won't do. So after actually fiddling with the DB settings and a restart, things looked like they improved. The load average was hanging out at around 0.8 for much of the time that I was watching it. Yay!
No, boo! Of course that's not the end of this story.
Taking a peek at things a few days later, we were back to square one. Load averages were jumping up to 5 or 6; sometimes into the teens. Yup, same issue as before. Once a few controllers were launched, the database would run a little slower, more processes would be launched before the first can complete, more processes, more slowness, until we have a real back load of processes waiting to complete. I was envisioning BanLink objects sending query after query until all of them in-world had been queued up and we've got 40+ processes waiting for a response from the server, which is now running so slow it can't even display this blog entry. So I did something delightfully simple…and maybe a little bit evil. I changed the BanLink controller just ever so slightly. It now decides whether or not it should run. Ooh, artificial intelligence, you say? Well I will venture to say that this is smarter than many people who don't know when to quit when they're ahead, but I don't expect it will be sending out an army of robots to take over the world just yet. All it does is check the server load. If it's just too darned high, it quits. Simple.
"But," you say, "what about the systems that are running and a griefer shows up and starts causing trouble? What about that? How will they be banned or ejected with this system if it's under heavy load?" Well, honestly, it won't really make a whole lot of difference with this addition. All it *is* doing is making sure that the system runs nicely. In other words, you can read this blog because BanLink won't take up all the CPU time even if there's a lot going on. And will it change things if you've got a griefer and are waiting for them to be ejected? Not really. If the server was already under a heavy load, you'd have to wait five or six seconds for the process to complete, if you were lucky. Now all it does is bail out and wait for you to query the system again. Most BanLink systems are querying every 5-10 seconds, so there's not a lot of time loss. And with this new code in place, the load average hasn't really jumped up into the teens like it used to. Sure, there's the potential for a site to be lost in the mix, but this is a quick and dirty stop-gap until the next code release (which is coming, I promise).
Yay! Right? It is Yay time, isn't it?
No, not quite, but we're almost there.
Now we've got the DB running better than it was. We've also got BanLink running a little better than it was. Now, let's really look and see how things are performing. Last weekend I wrote a bit of code which allows me to take a look at what BanLink is really doing. I should've done this long ago, as it's really interesting to see. After putting this up, we found that we're getting over 60,000 BanLink queries from the in-world objects every day. This averages at a little over 40 queries every minute. That may not sound like a whole lot, but we're close to a query per second. And honestly, at night, it's very much that. But the stats went to show a little more info that was useful. For instance, the average server load during the 24 hour period was over 2.5. Not quite what I'd been hoping for. Also, the average execution time for the BanLink controller is over 3.5 seconds. Yikes!
Alright, time for one more round of optimizations.
This time I went a bit deeper into the DB optimizations. It might be a bit more memory intensive, but I think it's needed at this point. I added a few additional tweaks which I hadn't thrown in there before. Next was to see if we could tune PHP just a bit. This time, it was up to eAccelerator. I'd thought about installing it a while ago, but hadn't quite gone the distance. eAccelerator is an optimizer for PHP which simply caches the PHP code in its compiled state so that each subsequent run doesn't have to do the compile. Sounds simple enough. Kind of stupidly simple concept, in fact. But it's not what PHP does by default. I'm a pretty big PHP advocate, but that really is just silly. So after a bit of time installing that, tweaking the OS just a tad, and letting it loose, we saw some major improvements. Load average now sits at 0.5 and the average execution time is down to 1 second. There's still a bit of inconsistency, as I'll see the average execution time a low as 0.2 seconds. I'd like to see 0.2 seconds as the average, but I guess this leaves room for those improvements that I hope to be getting to in the near future.
So is it Yay time? Yes, my friends, it is now Yay time. The servers are purple.