Archive for the ‘foo, bar and h4xx1ng’ Category

Twittipy 0.2

Friday, September 19th, 2008

I’m on a very brief break and hacked on my Twitter notifier for a couple of minutes. To tell you the truth, a couple of bugs have been ironed out a week after I posted the first version. It has been working so well, I didn’t bother playing around with the code.

Twittipy now prompts for your username and password. If you want it to save to the config file, just uncomment some of the lines. I’m not comfortable with the password saved in a file and I haven’t thought of a way to encrypt them. By the way, Pidgin saves passwords in plain text too. The login prompt is old code from a wxPython experiment some time ago (and it ain’t pretty).

After you get authenticated, an icon will appear in your system tray. To exit Twittipy, right-click on the icon.

Download it here.

Twittipy: A pynotify and pycurl experiment

Friday, June 27th, 2008

It’s been a drag since Twitter IM went down. I had a very brief “play” time the other night that gave birth to Twittipy. It’s a Twitter notifier written in python with pynotify and pycurl.

Running Twittipy

  1. Create a configuration file in your home directory (~/.twittipy)
  2. Set your Twitter username/email and password in the configuration file.
    Example:
    [general]
    username = johnsmith
    password = unhackable
  3. Make Twittipy executable
    $ chmod u+x twittipy.py
  4. Fire it up!
    $ ./twittipy.py &

Twittipy requires Python 2.5, pycurl and pynotify. If it complains of missing modules, you probably didn’t meet the requirements.

Roadmap

I did some googling and found other stuff to try that could improve Twittipy.

  • Port for other platforms (Windows with pywin32 and KDE via wxPython)
  • Icon at the notification area or system tray (via wxPython)
  • Encrypt or obfuscate password in config file and/or if password is not found in the config file, prompt and store in memory.
  • Make use of the other methods of the Twitter API.

Download it here. There are two more options in the config file, update_interval and last_update. The former defaults to 4 minutes and the latter to the current time.

I’m quite satisfied with it. The notifications of pynotify are less obtrusive than an IM message. There’s one noticeable quirk where a tweet repeats. We’ll fix that for the next release.

Comments are welcome and please check out tweetyPy as well (not mine but also an academic project).

I think about queues all the time.

Thursday, April 24th, 2008

Here’s something to waste your time and bore your brains out.

There are three nodes, say Nodes A, B and C. Node A and C are application servers. Node B is a messenger for nodes A and C. Although Node B is not needed but for reasons I’m not allowed to discuss and would rather not discuss, it has to be there and is the root cause why we have a problem.

Let’s start with an example series of events. A new transaction begins from Node C which sends data to Node B. Node B stores the data in a queue. Node A fetches x items from the queue every y seconds.

After Node A processes the transaction, it begins a new transaction. Node A sends data to Node B only this time, it is no longer stored in a queue. Node B immediately forwards the data to Node C.

On some occasions, Node C starts a new transaction as an acknowledgment from the data that originated from Node A. This transaction is treated as a transaction similar to the first one only that Node A knows that it’s an acknowledgment.

Originally, Node A and Node C communicated to each other directly. The queue didn’t exist. The queue came in to fix a couple of problems but it introduced new ones. Recently, the script in Node A that polls data from Node B died and now it’s catching up with backlog. The script processed data one by one. This showed how slow the main application in Node A is.

I already have a solution in mind and I enjoy writing boring stuff but I’d like to hear other views.

Insert to Asterisk’s Queue Log When a Member Is Called

Friday, April 4th, 2008

Asterisk’s queue_log can come in handy in many cases (queues in Asterisk can be called ACD). There are systems that go as far as depending solely on the queue_log to operate (including mine).

Every now and then requests requirements for a feature to tell if a call in the queue is being transferred to a queue member (an agent) comes up. I had a workaround that used an AGI script but it didn’t work all the time. I’ve been digging around the source code so that it’s inserted into queue_log instead. After three attempts of hacking one friggin’ line of C code, I finally did it. My sincerest apologies to my COMPRO1 and COMPRO2 professors.

You may get the patch for asterisk-1.4.19 over here.

P.S. I know that watching the queue_log isn’t much of a good idea. I only stole the concept and I wasn’t aware of AMI at that time. Migration plans are on the way.

myssh 0.2

Wednesday, January 2nd, 2008

Finally got the motivation to port this to Python. I don’t want to go through everything again so just read the original post. I also added myssh_print and myssh_flush. The former, if not provided with a label, prints all the profiles. The latter removes a profile.

Download (myssh_0_2.tar.gz 8.1K)

MySQL Woes

Tuesday, July 31st, 2007

I was having problems with slow queries last night and I noticed that MySQL, even with two other CPUs available, just stuck to one processor. I did a little googling and found out that MySQL is not that multi-threaded.

I’m currently working with about 70 instances, all versions, different configs and settings, different loads. (Sometimes more than 7000qps)

Basically, here are the facts:

MySQL relies on NTPL or PTHREADS to handle it’s concurrent processing, and this in itself is a design flaw. Real transactional databases use multiple processes and multiple threads within those processes. Using linux threads can push a small percentage of the load to other processors/cores but it’s not anywhere near what a real multi-process application can do

If you’re lucky, your top will have the “H” option which shows threads as fake separate processes. If you’ve got enough connections per second, you’ll see mysql throw off a few threads to a different processor, but your MPSTAT will show that only one of the cores is doing most of the work (based on interupts per second, which is more accurate than load average or any other variable in top) In fact, the majority of the load on the other cores comes from the operating system services (kjournald, syslogd, …)

MySQL was designed as a simple single-process database app that sits on the same server as php and apache, so if you’re running a typical LAMP blog site or zencart on one server, this is more than enough

If you install PostgreSQL, Oracle, Sybase, basically any real database application, you’ll see dramatic difference in how the workload is distributed. The CPU cores do about the same number of interupts per second

I’ve tested mysql on every platform, and the outcome is always the same. A Sun T2000 box with 32 cores, regardless of mysql config, thread library, or version, will show activity on only one core from MySQL (even with hundreds of open connections)

Anything more than a single core CPU is completely useless for a database server running mysql. If you want to spend money on what really matters, get a decent fiber storage array with a fast controller

The conclusion: If you want to leverage multiple CPUs on a server running nothing but a database application, don’t use MySQL. Otherwise get a single core pizzabox and a fast disk controller and you’re all set

If this is true, can I setup another instance of MySQL on the same machine and have it replicate?

I listened to the July 21st episode of FLOSS Weekly the other night and the topic was Postgresql. This is exactly what Josh Berkus was talking about. I should try playing with Postgresql again. Table partitioning sounds like a huge lifesaver and I could sure use the extra data types.

Google, on the other hand, uses MySQL in a couple of apps. They use a huge cluster of mid-range machines with large memory. I’m after utilizing the other available CPUs.

Links

Bad Apples (Cont’d)

Friday, July 20th, 2007

First off, iPodLinux won’t install on my iPod. It just keeps failing. I’m heart broken. Second, can’t Apple screw the fancy design for practical use? I always have a hard time unplugging the USB cable.

I’m still not in the mood to upload my songs. I’m putting up with Apache’s indexing until I get MythTV installed. My GNUMP3d installation is messed up. The last 5 seconds of the tracks are clipped off. I’d try to hack it but I’m no PERL genius.

How to Install Munin on CEntOS

Thursday, June 14th, 2007

Munin is a monitoring tool for servers. It uses RRDtool to log and graph data from your servers. The plugin API is very easy to grasp. Actually, I haven’t read the API documentation yet. I just looked at the output of the plugins and it looks easy to achieve. The data can be accessed through the web.

This guide will walk you through installing and configuring Munin on CEntOS 4.3/4.4 x86. The steps are pretty much the same for later releases of CEntOS, Red Hat, Fedora [Core] and/or Red Hat-based installations.

Munin works by polling your servers for the data hence two applications, Munin and Munin Node. The former periodically gathers data (cronned) and the latter serves the data to the former. Please refer to the following for our example configuration. You can make up a domain if you want Munin to group your servers similar to the live demo.

Munin “Graph Server” - alpha.sample.net (192.168.1.1)
A Munin Node - bravo.sample.net (192.168.1.2)

1. Installing and Configuring Munin

In this section, we set it up on alpha.sample.net

  1. Add the RPMforge repository.
    rpm -Uhv http://apt.sw.be/packages/rpmforge-release/rpmforge-release-0.3.6-1.el4.rf.i386.rpm
    This step is optional if your Linux distribution has the packages in its default repositories. For other versions and architectures, click here.
  2. Install munin.
    yum -y install munin

  3. Change the ownership of the Munin web docroot to munin.
    chown -R munin:munin /var/www/munin
  4. The default configuration file (in version 1.2.5-1) the value for the web docroot points to the wrong directory. Replace the value of htmldir from /var/www/html/munin to /var/www/munin at /etc/munin/munin.conf (line 7).
  5. Restart Apache and Cron
    service httpd restart && service crond restart
  6. You can check if it’s working through your browser (i.e. http://192.168.1.1/munin/). You will get a 404 (not found) if you don’t supply a trailing slash.

2. Add a Node

In this section, we will configure bravo.sample.net.

  1. Add the RPMforge repository (see 1.1).
  2. Install Munin Node.
    yum -y install munin-node
  3. Configure. Edit /etc/munin/munin-node.conf with your favorite text editor.
    • Allow the graph server (alpha.sample.net/192.168.1.1) to poll the node.
      allow ^192\.168\.1\.1$
    • If your server doesn’t report the correct hostname, add the following line
      host_name bravo.sample.net
    • If your servers have two interfaces and on the same LAN (e.g. one for Internet and another for LAN), you can configure the node to bind and listen on the local interface by changing the value of host (line 13) from * to the local IP of the node.
  4. Start munin-node and set to start on bootup.
    service munin-node start
    chkconfig munin-node on
  5. Edit Munin’s configuration on the graph server (/etc/munin/munin.conf).
    [bravo.sample.net]
    address 192.168.1.2
    use_node_name yes

Wait for at least 5 minutes for the new node to appear. You can also install the node on the graph server. The default node configuration will work out of the box.

3. Install/Activate Some Plugins

This section should familiarize you with the plugin installation routine. Plugins are installed in the nodes.

  • Apache
    1. Create a symbolic link to the Apache plugins (stored in /usr/share/munin/plugins) in the plugin folder.
      ln -s /usr/share/munin/plugins/apache_* /etc/munin/plugins/

    2. Enable server status reports. Add the following to Apache’s configuration file.
      ExtendedStatus On
      <Location /server-status>
      SetHandler server-status
      Order Deny,Allow
      Deny from all
      Allow from 127.0.0.1
      </Location>
    3. Restart Apache and the node
      service httpd restart && service munin-node restart
  • Asterisk
    1. Download the plugins for your Asterisk version from here.
    2. Extract them to /usr/share/munin/plugins.
    3. Make the files executable
      chmod 755 /usr/share/munin/plugins/asterisk_*

    4. Configure Asterisk Manager by adding/changing the following in /etc/asterisk/manager.conf
      [general]
      enabled = yes
      port = 5038

      [munin]
      secret = somepassword
      permit = 127.0.0.1 ;if this doesn’t work, use the local IP
      write = system,call,log,verbose,command,agent,user

    5. Add the following to the plugin configuration file in /etc/munin/plugin-conf.d/munin-node
      [asterisk_*]
      env.username munin
      env.secret somepassword
    6. Reload Asterisk’s configuration and restart the node.
      asterisk -rx reload >> /dev/null && service munin-node restart
  • MySQL
    1. Create a symbolic link to the MySQL plugins (stored in /usr/share/munin/plugins) in the plugin folder.
      ln -s /usr/share/munin/plugins/mysql_* /etc/munin/plugins/
    2. If your root user has a password (or want to use a different user), edit the plugin configuration file in /etc/munin/plugin-conf.d/munin-node and uncomment line 16 by removing the leading hash (#). Then change the parameters that will be used when mysqladmin is run.
    3. Restart the node
      service munin-node restart
  • MTR
    1. Make sure you have the latest version of MTR.
      yum -y install mtr && yum -y update mtr
    2. Download the plugin here (direct link)
    3. Extract to /usr/share/munin/plugins
    4. Make the file executable.
      chmod 755 /usr/share/munin/plugins/mtr100_
    5. Create a symbolic link to the plugin (stored in /usr/share/munin/plugins) in the plugin folder. Append the host that you want to query to the link of the name.
      ln -s /usr/share/munin/plugins/mtr100_ /etc/munin/plugins/mtr100_somehost.com
    6. To add another host to query, just create another symbolic link.
    7. Add the following to the plugin configuration file in /etc/munin/plugin-conf.d/munin-node
      [mtr100_*]
      timeout 60
    8. Restart the node
      service munin-node restart

4. Links

Trackback: The MSFT Fanboys Are Here

Thursday, May 17th, 2007

This is a response to the InformationWeek article, “Why Doesn’t Microsoft Have A Cult Religion?” that Ars Technica also responded to.

I may be a Linux fanboy but I “worked” for Microsoft before and they do have a cult following. I don’t know how it is in the US but here in the Philippines, they have a very active community. In fact, most of my classmates wanted to study .NET for its ease of use, short learning curve, seamless integration, nice IDE and its popularity.

Unfortunately for them, my powers were strong enough to force PHP as the language for the project (supposed to be Java). They must hate me now. I wonder how Ash is doing at IBM.

I’m grateful enough to have been in the company of Jasper, Microsoft’s largest fanboy in the Philippines. He’s president of the Philippine .NET Users’ Group (PHINUG). Just so you know, Microsoft [Philippines] has a lot of evangelists and if I’m not mistaken there are more MSFT organizations.

I don’t only know a tech evangelist but I also encountered an MCSE that really looked way down on Linux as a hard-to-maintain enterprise OS. This guy was the chief software architect or the big kahuna of a large travel agency here. Their development team is paid on a per-project basis. Their charges are ridiculously overpriced that other departments prefer to outsource most of the projects.

Not to mention that he likes his black Technet jacket. But to be fair. I also like to wear the Polgas shirt and the theSpoke laptop bag is my only bag.

Think about it. When was the last time an editor was fired because of a scathing article entitled, “10 Things We Hate About Microsoft?” When was the last time a group of developers stood up at a VS Live show and shouted … “Yea, man! Orcas Rocks! Language Integrated Query is da’ Bomb! New and improved ADO.Net? Oh, no you didn’t!” It just doesn’t happen.

I love this paragraph but I saw this happen once or twice. I was amused at some level the first time VS2005 was demonstrated but I’m not the WYSIWYG type of guy.

My guesstimate is that 4 in 5 developers I know are more knowledgeable at .NET than FOSS platforms. As for Xbox, zune and Vista fans, I still have to meet them.

Just Like Everyone Else

Friday, May 4th, 2007

Just in case the evil MPAA succeeds in shutting down every blog that posted the hex code, I’m posting it like everyone else is.

09 F9 11 02 9d 74 E8 5b D8 41 56 C5 63 56 88 C0

There’s even a music video. Two domains for the code (2). You can also get shirts from here.

	// Processing Key
	static unsigned char processing_key[16] = {0x09,0xF9,0x11,0x02,0x9D,0x74,0xE3,0x5B,0xD8,0x41,0x56,0xC5,0x63,0x56,0x88,0xC0};

	// Encrypted C Value
	static unsigned char encrypted_c_value[16] = {0x6D,0x02,0xCA,0xC6,0x7B,0x1A,0x7E,0x95,0xC2,0x16,0xEF,0xD4,0xC9,0x28,0x09,0xCF};

	//Decrypted C Value
	static unsigned char decrypted_c_value[16];
	static unsigned char uv[4] = {0x00,0x00,0x00,0x01};

	// Media Key
	static unsigned char media_key[16]; 

	//Encrypted Verification Data (King Kong)
	static unsigned char encrypted_verification_data[16] = {0x87,0xB8,0xA2,0xB7,0xC1,0x0B,0x9F,0xAD,0xF8,0xC4,0x36,0x1E,0x23,0x86,0x59,0xE5};

	//Decrypted Verification Data Should Be
	static unsigned char decrypted_verification_data_should_be[8] = {0x01,0x23,0x45,0x67,0x89,0xAB,0xCD,0xEF};

	//Decrypted Verification Data
	static unsigned char decrypted_verification_data[16];

	// Volume ID
	static unsigned char volume_id[16] = {0x40,0x00,0x09,0x18,0x20,0x06,0x08,0x41,0x00,0x20,0x20,0x20,0x20,0x20,0x00,0x00};

	//Decrypted Volume ID
	static unsigned char decrypted_volumeid[16];

	//Volume Unique Key
	static unsigned char volume_unqiue_key[16];


	// First decrypt the C-value with the processing key
	oRijndael.MakeKey((char *)processing_key, CRijndael::sm_chain0, 16, 16);
	oRijndael.DecryptBlock((char *)encrypted_c_value, (char *)decrypted_c_value);

	// Then XOR it with with the uv (of the corresponding C-value)
	for (j = 0; j < 16; j++)
	{
		if (j < 12)
		{
			media_key[j] = decrypted_c_value[j];
		}
		else
		{
			media_key[j] = decrypted_c_value[j]^uv[j-12];
		}
	}

	// Then check if the resulting media key is correct using the verify media key record
	oRijndael.MakeKey((char *)media_key, CRijndael::sm_chain0, 16, 16);
	oRijndael.DecryptBlock((char *)encrypted_verification_data, (char *)decrypted_verification_data);

	if (!memcmp(decrypted_verification_data_should_be, decrypted_verification_data, 8))
	{
		for (j = 0; j < 16; j++)
		{
			printf("%02X ", decrypted_verification_data[j]);
		}
	}
	printf("\n");

	// Then do a AES-G (basicly a decrypt and an XOR) on the media key + volumeID
	oRijndael.MakeKey((char *)media_key, CRijndael::sm_chain0, 16, 16);
	oRijndael.DecryptBlock((char *)volume_id, (char *)decrypted_volumeid);
	for (j = 0; j < 16; j++)
	{
		volume_unqiue_key[j] = volume_id[j]^decrypted_volumeid[j];
	}
	printf("\n");

	// This results in the Volume Unique Key
	for (j = 0; j < 16; j++)
	{
		printf("%02X ", volume_unqiue_key[j]);
	}
	printf("\n");

via doom9 forums