iRant » Software/Code

Archive for the ‘Software/Code’ Category

MySQL Woes

Tuesday, July 31st, 2007

I was having problems with slow queries last night and I noticed that MySQL, even with two other CPUs available, just stuck to one processor. I did a little googling and found out that MySQL is not that multi-threaded.

I’m currently working with about 70 instances, all versions, different configs and settings, different loads. (Sometimes more than 7000qps)

Basically, here are the facts:

MySQL relies on NTPL or PTHREADS to handle it’s concurrent processing, and this in itself is a design flaw. Real transactional databases use multiple processes and multiple threads within those processes. Using linux threads can push a small percentage of the load to other processors/cores but it’s not anywhere near what a real multi-process application can do

If you’re lucky, your top will have the “H” option which shows threads as fake separate processes. If you’ve got enough connections per second, you’ll see mysql throw off a few threads to a different processor, but your MPSTAT will show that only one of the cores is doing most of the work (based on interupts per second, which is more accurate than load average or any other variable in top) In fact, the majority of the load on the other cores comes from the operating system services (kjournald, syslogd, …)

MySQL was designed as a simple single-process database app that sits on the same server as php and apache, so if you’re running a typical LAMP blog site or zencart on one server, this is more than enough

If you install PostgreSQL, Oracle, Sybase, basically any real database application, you’ll see dramatic difference in how the workload is distributed. The CPU cores do about the same number of interupts per second

I’ve tested mysql on every platform, and the outcome is always the same. A Sun T2000 box with 32 cores, regardless of mysql config, thread library, or version, will show activity on only one core from MySQL (even with hundreds of open connections)

Anything more than a single core CPU is completely useless for a database server running mysql. If you want to spend money on what really matters, get a decent fiber storage array with a fast controller

The conclusion: If you want to leverage multiple CPUs on a server running nothing but a database application, don’t use MySQL. Otherwise get a single core pizzabox and a fast disk controller and you’re all set

If this is true, can I setup another instance of MySQL on the same machine and have it replicate?

I listened to the July 21st episode of FLOSS Weekly the other night and the topic was Postgresql. This is exactly what Josh Berkus was talking about. I should try playing with Postgresql again. Table partitioning sounds like a huge lifesaver and I could sure use the extra data types.

Google, on the other hand, uses MySQL in a couple of apps. They use a huge cluster of mid-range machines with large memory. I’m after utilizing the other available CPUs.

Bad Apples (Cont’d)

Friday, July 20th, 2007

First off, iPodLinux won’t install on my iPod. It just keeps failing. I’m heart broken. Second, can’t Apple screw the fancy design for practical use? I always have a hard time unplugging the USB cable.

I’m still not in the mood to upload my songs. I’m putting up with Apache’s indexing until I get MythTV installed. My GNUMP3d installation is messed up. The last 5 seconds of the tracks are clipped off. I’d try to hack it but I’m no PERL genius.

Posted in Geektraptions, Software/Code, Techitude, foo, bar and h4xx1ng | Comments

DZone

Tuesday, June 19th, 2007

Somebody just added one of my entries to DZone. It’s like Digg for developers.

DZone is a free link-sharing community for developers

anyone can submit new links to the incoming queue

members vote on upcoming links to determine what gets promoted

everyone can browse, search and comment on links

Posted in Software/Code, Techitude, The Web/Internet | Comments

How to Install Munin on CEntOS

Thursday, June 14th, 2007

Munin is a monitoring tool for servers. It uses RRDtool to log and graph data from your servers. The plugin API is very easy to grasp. Actually, I haven’t read the API documentation yet. I just looked at the output of the plugins and it looks easy to achieve. The data can be accessed through the web.

This guide will walk you through installing and configuring Munin on CEntOS 4.3/4.4 x86. The steps are pretty much the same for later releases of CEntOS, Red Hat, Fedora [Core] and/or Red Hat-based installations.

Munin works by polling your servers for the data hence two applications, Munin and Munin Node. The former periodically gathers data (cronned) and the latter serves the data to the former. Please refer to the following for our example configuration. You can make up a domain if you want Munin to group your servers similar to the live demo.

Munin “Graph Server” - alpha.sample.net (192.168.1.1)
A Munin Node - bravo.sample.net (192.168.1.2)

1. Installing and Configuring Munin

In this section, we set it up on alpha.sample.net

Add the RPMforge repository.
rpm -Uhv http://apt.sw.be/packages/rpmforge-release/rpmforge-release-0.3.6-1.el4.rf.i386.rpm
This step is optional if your Linux distribution has the packages in its default repositories. For other versions and architectures, click here.
Install munin.
yum -y install munin
Change the ownership of the Munin web docroot to munin.
chown -R munin:munin /var/www/munin
The default configuration file (in version 1.2.5-1) the value for the web docroot points to the wrong directory. Replace the value of htmldir from /var/www/html/munin to /var/www/munin at /etc/munin/munin.conf (line 7).
Restart Apache and Cron
service httpd restart && service crond restart
You can check if it’s working through your browser (i.e. http://192.168.1.1/munin/). You will get a 404 (not found) if you don’t supply a trailing slash.

2. Add a Node

In this section, we will configure bravo.sample.net.

Add the RPMforge repository (see 1.1).
Install Munin Node.
yum -y install munin-node
Configure. Edit /etc/munin/munin-node.conf with your favorite text editor.
- Allow the graph server (alpha.sample.net/192.168.1.1) to poll the node.
  allow ^192\.168\.1\.1$
- If your server doesn’t report the correct hostname, add the following line
  host_name bravo.sample.net
- If your servers have two interfaces and on the same LAN (e.g. one for Internet and another for LAN), you can configure the node to bind and listen on the local interface by changing the value of host (line 13) from * to the local IP of the node.
Start munin-node and set to start on bootup.
service munin-node start chkconfig munin-node on
Edit Munin’s configuration on the graph server (/etc/munin/munin.conf).
[bravo.sample.net] address 192.168.1.2 use_node_name yes

Wait for at least 5 minutes for the new node to appear. You can also install the node on the graph server. The default node configuration will work out of the box.

3. Install/Activate Some Plugins

This section should familiarize you with the plugin installation routine. Plugins are installed in the nodes.

Apache
1. Create a symbolic link to the Apache plugins (stored in /usr/share/munin/plugins) in the plugin folder.
  ln -s /usr/share/munin/plugins/apache_* /etc/munin/plugins/
2. Enable server status reports. Add the following to Apache’s configuration file.
  ExtendedStatus On <Location /server-status> SetHandler server-status Order Deny,Allow Deny from all Allow from 127.0.0.1 </Location>
3. Restart Apache and the node
  service httpd restart && service munin-node restart
Asterisk
1. Download the plugins for your Asterisk version from here.
2. Extract them to /usr/share/munin/plugins.
3. Make the files executable
  chmod 755 /usr/share/munin/plugins/asterisk_*
4. Configure Asterisk Manager by adding/changing the following in /etc/asterisk/manager.conf
  [general] enabled = yes port = 5038 [munin] secret = somepassword permit = 127.0.0.1 ;if this doesn’t work, use the local IP write = system,call,log,verbose,command,agent,user


MySQL

Create a symbolic link to the MySQL plugins (stored in /usr/share/munin/plugins) in the plugin folder.

ln -s /usr/share/munin/plugins/mysql_* /etc/munin/plugins/
If your root user has a password (or want to use a different user), edit the plugin configuration file in /etc/munin/plugin-conf.d/munin-node and uncomment line 16 by removing the leading hash (#). Then change the parameters that will be used when mysqladmin is run.
Restart the node

service munin-node restart


MTR

Make sure you have the latest version of MTR.

yum -y install mtr && yum -y update mtr
Download the plugin here (direct link)
Extract to /usr/share/munin/plugins
Make the file executable.

chmod 755 /usr/share/munin/plugins/mtr100_
Create a symbolic link to the plugin (stored in /usr/share/munin/plugins) in the plugin folder. Append the host that you want to query to the link of the name.

ln -s /usr/share/munin/plugins/mtr100_ /etc/munin/plugins/mtr100_somehost.com
To add another host to query, just create another symbolic link.
Add the following to the plugin configuration file in /etc/munin/plugin-conf.d/munin-node

[mtr100_*]

timeout 60
Restart the node

service munin-node restart


4. Links

How to install on Debian machines
How to Write Munin Plugins
Troubleshooting Munin
MuninExchange - Munin Plugin Repository



				 Posted in Open Source,  Software/Code,  Techitude,  foo, bar and h4xx1ng |   Comments



				
				The Long Wait is Over …Almost
				Monday, May 21st, 2007

				
					I’m pretty sure you’ve read about the Starcraft II announcement by now. The CGI trailer is amazing. Heyl. It sure is about toim.
This is also the first time where I’m using a gameplay screen cap as a wallpaper. The graphics are amazing! Like other geeks and gamers on the planet, I’m dehydrated due to excessive drooling and I’m still pissing on my pants.



click to embiggen
But don’t get that excited yet. You have to see the preview of Starcraft III!


Here are some gameplay footage (via cow).
				

				 Posted in Software/Code,  Techitude |   Comments

			

				
				Trackback: The MSFT Fanboys Are Here
				Thursday, May 17th, 2007

				
					This is a response to the InformationWeek article, “Why Doesn’t Microsoft Have A Cult Religion?” that Ars Technica also responded to.
I may be a Linux fanboy but I “worked” for Microsoft before and they do have a cult following. I don’t know how it is in the US but here in the Philippines, they have a very active community. In fact, most of my classmates wanted to study .NET for its ease of use, short learning curve, seamless integration, nice IDE and its popularity.
Unfortunately for them, my powers were strong enough to force PHP as the language for the project (supposed to be Java). They must hate me now. I wonder how Ash is doing at IBM.
I’m grateful enough to have been in the company of Jasper, Microsoft’s largest fanboy in the Philippines. He’s president of the Philippine .NET Users’ Group (PHINUG). Just so you know, Microsoft [Philippines] has a lot of evangelists and if I’m not mistaken there are more MSFT organizations.
I don’t only know a tech evangelist but I also encountered an MCSE that really looked way down on Linux as a hard-to-maintain enterprise OS. This guy was the chief software architect or the big kahuna of a large travel agency here. Their development team is paid on a per-project basis. Their charges are ridiculously overpriced that other departments prefer to outsource most of the projects.
Not to mention that he likes his black Technet jacket. But to be fair. I also like to wear the Polgas shirt and the theSpoke laptop bag is my only bag.
Think about it. When was the last time an editor was fired because of a scathing article entitled, “10 Things We Hate About Microsoft?” When was the last time a group of developers stood up at a VS Live show and shouted … “Yea, man! Orcas Rocks! Language Integrated Query is da’ Bomb! New and improved ADO.Net? Oh, no you didn’t!” It just doesn’t happen.
I love this paragraph but I saw this happen once or twice. I was amused at some level the first time VS2005 was demonstrated but I’m not the WYSIWYG type of guy.
My guesstimate is that 4 in 5 developers I know are more knowledgeable at .NET than FOSS platforms. As for Xbox, zune and Vista fans, I still have to meet them.
				

				 Posted in Software/Code,  Techitude,  foo, bar and h4xx1ng |   Comments

			

				
				Do Yourself and Others A Favor
				Monday, May 14th, 2007

				
					If you have that annoying Yahoo! IM virus, may I suggest that you log out of it and use meebo or another IM client? It’s really annoying. I even told a couple of “victims” to turn off their computer. Either that or you get blocked.
If you’re hardworking enough, here’s what you can do:

Try Arun Vishnu’s Fix
Read this article from Microsoft
Get better anti-virus software
Or switch to a better OS and never experience it again. Ever.

				

				 Posted in Software/Code,  Techitude |   Comments

			

				
				Introduction to Asterisk (via Systm)
				Wednesday, May 9th, 2007

				
					
Download it from here.
				

				 Posted in Open Source,  Software/Code,  Techitude |   Comments

			

				
				Just Like Everyone Else
				Friday, May 4th, 2007

				
					Just in case the evil MPAA succeeds in shutting down every blog that posted the hex code, I’m posting it like everyone else is.
09 F9 11 02 9d 74 E8 5b D8 41 56 C5 63 56 88 C0
There’s even a music video. Two domains for the code (2). You can also get shirts from here.


	// Processing Key
	static unsigned char processing_key[16] = {0x09,0xF9,0x11,0x02,0x9D,0x74,0xE3,0x5B,0xD8,0x41,0x56,0xC5,0x63,0x56,0x88,0xC0};

	// Encrypted C Value
	static unsigned char encrypted_c_value[16] = {0x6D,0x02,0xCA,0xC6,0x7B,0x1A,0x7E,0x95,0xC2,0x16,0xEF,0xD4,0xC9,0x28,0x09,0xCF};

	//Decrypted C Value
	static unsigned char decrypted_c_value[16];
	static unsigned char uv[4] = {0x00,0x00,0x00,0x01};

	// Media Key
	static unsigned char media_key[16]; 

	//Encrypted Verification Data (King Kong)
	static unsigned char encrypted_verification_data[16] = {0x87,0xB8,0xA2,0xB7,0xC1,0x0B,0x9F,0xAD,0xF8,0xC4,0x36,0x1E,0x23,0x86,0x59,0xE5};

	//Decrypted Verification Data Should Be
	static unsigned char decrypted_verification_data_should_be[8] = {0x01,0x23,0x45,0x67,0x89,0xAB,0xCD,0xEF};

	//Decrypted Verification Data
	static unsigned char decrypted_verification_data[16];

	// Volume ID
	static unsigned char volume_id[16] = {0x40,0x00,0x09,0x18,0x20,0x06,0x08,0x41,0x00,0x20,0x20,0x20,0x20,0x20,0x00,0x00};

	//Decrypted Volume ID
	static unsigned char decrypted_volumeid[16];

	//Volume Unique Key
	static unsigned char volume_unqiue_key[16];


	// First decrypt the C-value with the processing key
	oRijndael.MakeKey((char *)processing_key, CRijndael::sm_chain0, 16, 16);
	oRijndael.DecryptBlock((char *)encrypted_c_value, (char *)decrypted_c_value);

	// Then XOR it with with the uv (of the corresponding C-value)
	for (j = 0; j < 16; j++)
	{
		if (j < 12)
		{
			media_key[j] = decrypted_c_value[j];
		}
		else
		{
			media_key[j] = decrypted_c_value[j]^uv[j-12];
		}
	}

	// Then check if the resulting media key is correct using the verify media key record
	oRijndael.MakeKey((char *)media_key, CRijndael::sm_chain0, 16, 16);
	oRijndael.DecryptBlock((char *)encrypted_verification_data, (char *)decrypted_verification_data);

	if (!memcmp(decrypted_verification_data_should_be, decrypted_verification_data, 8))
	{
		for (j = 0; j < 16; j++)
		{
			printf("%02X ", decrypted_verification_data[j]);
		}
	}
	printf("\n");

	// Then do a AES-G (basicly a decrypt and an XOR) on the media key + volumeID
	oRijndael.MakeKey((char *)media_key, CRijndael::sm_chain0, 16, 16);
	oRijndael.DecryptBlock((char *)volume_id, (char *)decrypted_volumeid);
	for (j = 0; j < 16; j++)
	{
		volume_unqiue_key[j] = volume_id[j]^decrypted_volumeid[j];
	}
	printf("\n");

	// This results in the Volume Unique Key
	for (j = 0; j < 16; j++)
	{
		printf("%02X ", volume_unqiue_key[j]);
	}
	printf("\n");

via doom9 forums


				


				 Posted in Software/Code,  Techitude,  foo, bar and h4xx1ng |   Comments

			


				
				Following the XML Model
				Sunday, April 29th, 2007

				
					Last week, I tested one of the parts of my pet project. To be more precise, it was a stress test. The result wasn’t so bad. Through this test, I found out which parts can make the server bog down.
The application is simple. It’s a page that shows useful information to an inbound call center agent and a menu for call dispositions. It shows session data and statistics, queue information, call details and a list of call dispositions that they can choose. In short, it’s a helper screen with a touch of CRM.
The test was done on about 150 seats (I think). I think at production’s peak hours, it makes 26 queries per user every 5 seconds (8 of those are unchangeable because of authentication). So that should be about 780 querries per second. Those queries are called via AJAX.
Although that didn’t cause the server to crash, it reached the maximum number of users logged in at the database (using persistent connections).
To rectify it, I can increase that limit. But only losers do that. I’ll try some form of caching. I don’t know why I didn’t think of it before. My plan is to execute the 2 queries that are repeated every X seconds (call and queue data) and cache the results. That should reduce the load on the database. The session data can be event triggered, reducing it even further.
The cache will contain three directories at its root (S, C and Q)*. Then each directory should have the files containing the results in a file named with the key in the where clause. The format of the contents should be something similar to JSON (XML just takes too much work).
So when the AJAX request is sent, all it has to do then is get that file and print its contents (the client-side script parses that data already). At that point, the number of queries per second would’ve been reduced to a near-constant rate of 2 (excluding authentication).
Wow, parang ALGOCOM na rin.
*I think McDonald’s used it too. I think it was “service cleanliness and quality”.
				

				 Posted in Software/Code,  Techitude,  foo, bar and h4xx1ng |   Comments

			

			
	

	
		
			« Older Entries
			Newer Entries »