Wednesday, June 1, 2011

Converting EXT3 partition into EXT4 on Ubuntu

EXT4 type partition is the successor of EXT3, but it may not be turned on by default installation. Indeed, EXT4 driver is backward compatible with EXT3/EXT2 partition. The conversion from EXT3 to EXT4 partition is straight forward.

Please backup important data and software you have been using on Ubuntu Linux.

BEFORE DOING ANYTHING, PLEASE SHUTDOWN ALL LAMPP RELATED PROCESSES:
$ sudo /opt/lampp/lampp stop
To switch the existing Ext3 file system to the new Ext4 driver, it is necessary to edit a configuration called "fstab":
$ sudo nano /etc/fstab
Look for ext3 on the line that defines your disk and change it to ext4.

To ensure the right device to be converted, you may check the name of the device by issuing the following command:
$ df -h

Make a note of the device as you will need it in the next step. In the example above, the device for the encrypted partition is "/dev/mapper/encrypted".

Next reboot your system. This is required so that you switch to the Ext4 Kernel driver. Do not continue without rebooting first.

Enable Ext4 features:
$ sudo tune2fs -O extents,uninit_bg,dir_index /dev/mapper/encrypted
WARNING: Once you run this command, the filesystem will no longer be mountable using the ext3 filesystem!

After running this command (specifically, after setting the uninit_bg parameter), it is also important to run fsck to fix up some on-disk structures that tune2fs has modified:
$ e2fsck -yfDC0 /dev/mapper/encrypted
Notes:

Running fsck will complain about "One or more block group descriptor checksums are invalid" - this is expected and one of the reasons why tune2fs requests to fsck.
By enabling the extents feature new files will be created in extents format, but this will not convert existing files to use extents. Non-extent files can be transparently read and written by Ext4.
If you convert your root filesystem ("/") to ext4, and you use the GRUB boot loader, you will need to install a version of GRUB which understands ext4. Your system may boot OK the first time, but when your kernel is upgraded, it will become unbootable (press Alt+F+F to check the filesystem).
If you do the conversion for the root fs on a live system you'll have to reboot for fsck to run safely. You might also need to add rootfstype=ext4 to the kernel's command line so the partition is not mounted as ext3.

WARNING:
It is NOT recommended to resize the inodes using resize2fs with e2fsprogs 1.41.0 or later, as this is known to corrupt some filesystems.
If you omit "uninit_bg" on the tunefs command, you can skip the fsck step.

Installing open-vm-tools on Ubuntu Guest VM

To help boost the performance of Ubuntu Guest VM, it is recommended to install VMware tools. The problem is it may not fully compatible with various release of Ubuntu. Ubuntu do have their own repository of Open VM tools for use. So, why not have a try on it?

Installing open-vm-tools from Ubuntu repository:

The VMware tools are part of open-vm-tools. Make sure that the "multiverse" repository is enabled and do:

Install kernel headers so modules will work
We need this on a 10.04 guest running in a Fusion 3 host
$ apt-get install linux-headers-virtual
Install kernel modules
$ apt-get install --no-install-recommends open-vm-dkms
EITHER: install tools for an xorg install
$ apt-get install open-vm-tools
OR: a headless install
$ apt-get install --no-install-recommends open-vm-tools

Direct Upgrade Ubuntu from 9.04 to 9.10

Ubuntu upgrade from 9.04 to 9.10 can never be easy like this:

Before the actions, it is essential to backup any running database and web server for disaster recovery and then turn them off completely.

You can easily upgrade over the network with the following procedure.

1. Start System/Administration/Update Manager

2. Click the Check button to check for new updates.

3. If there are any updates to install, use the Install Updates button to install them, and press Check again after that is complete.

4. A message will appear informing you of the availability of the new release.

5. Click Upgrade.

6. Follow the on-screen instructions.

Finally, reboot the system and see all changes.

There was compatibility issue regarding XAMPP package since people complained they couldn't get XAMPP based server startup after the reboot. In my case, it did happen before. In recent trial of GUI upgrade directly from Ubuntu Update Manager, the problem seems to be solved. I have even got another notice from Update Manager about Ubuntu upgrade to version 10.04 as well.

Tuesday, May 31, 2011

Resync MySQL Slave with Master

For various kinds of reasons, it is not unusual to find MySQL Slave instance in a broken state during the replication. The quick fix requires a perfect database backup of Master instance. From the backup file, two parameters must be extracted for restoration of data onto Slave Server:

bin-log filename: MASTER_LOG_FILE
bin-log position: MASTER_LOG_POS

Of course, the overall system performance may be affected during the backup operation. However, MySQL Master server can be kept running as normal without any interruption.

You may find it difficult to find all the switches required for a successful online backup operation. Fortunately, an example is here.

Before anything happens, you'll find it easier to manage all those by opening two separate terminal consoles for both Master and Slave instances.

Terminal console for Master server:

The following command (recommended to run under “sudo –s” environment) will dump all databases on Master instance:
$ mysqldump -uroot -p \-S /etc/mysql/mysql.sock \
--master-data --hex-blob --opt --single-transaction \
--comments --dump-date --no-autocommit \
--all-databases > target_backup.sql

Please make sure the path of socket file of MySQL Master server is correct.

With “--master-data” switch, Mysqldump program will make a comment for the bin-log filename and position which can be really useful for resynchronization afterwards.

With “--single-transaction” switch, a global lock is acquired for a very short time at the beginning of the dump while any action on logs happens at the exact moment of the dump.

With “--hex-blob” switch, image data with BLOB type can be well preserved inside the backup without any disruption.

Once finished, you'll get a perfect backup file from MySQL Master server named "target_backup.sql".

Terminal console for Slave server:

Now, it is time to put the data onto MySQL Slave server. Please beware that the socket file used here becomes “/etc/mysql2/mysql2.sock”. Don’t it mix up with that of Master instance.

$ mysql -uroot -p -S /etc/mysql2/mysql2.sock < target_backup.sql

Now, login to MySQL Slave server:

$ mysql –root –p –S /etc/mysql2/mysql2.sock


Please check if MySQL user role named “slaveuser” with “replication” privileges does exist on Slave server.

Then, execute the following commands to resynchronize Slave server:

mysql> STOP SLAVE;
mysql> RESET SLAVE;
mysql> CHANGE MASTER TO MASTER_HOST=’localhost’,
>MASTER_PORT=3306,
>MASTER_USER=’slaveuser’,
>MASTER_PASSWORD=’XXXXXXXX’,
>MASTER_LOG_FILE=’mysql-bin.XXXXXXXX’,
>MASTER_LOG_POS=XXXXX
>;
mysql> START SLAVE;

For the corresponding values of MASTER_LOG_FILE and MASTER_LOG_POS, please read through the heading comments inside “target_backup.sql” file.

To check the status, please issue the following command:

mysql> SHOW SLAVE STATUS;

It is useful to check the error log of MySQL Slave server to confirm whether everything is back on track.

Thursday, May 26, 2011

CeBIT Australia 2011

CeBIT 2011 is going to be held between 31 May 2011 and 2 June 2011. Both Technology Enthusiasts and Government Sectors are involved in this biggest event near Darling Harbour in Sydney. As I can imagine how many people will be there since last CeBIT event, it is definitely a good chance to meet top technologists and also listen to Julia about the new progress of the big plan for National Broadband Network.

Apart fromt the exhibition itself, there are upcoming hot topics to be discussed in the conferences. Major events include:
  • NBN Conference
  • Cloud Computing Conference
  • eGovernment Forum
  • eHealth Conference
  • Executive Briefing
  • Retail Conference
  • WebForward Conference
 Can't wait to be there and hopefully see you in the exhibition!

Sunday, May 22, 2011

PHP PDF generating over HTTPS issue on IE6 with blank page returned [REALLY SOVLED]

This is really a typical question raised at all time about mysterious IE6 PDF opening issue over HTTPS connection:


How can we successfully output PDF dynamically via HTTPS connection on various browser including IE6?


Whatever server-side language you use, you may encounter this yourself once or more in your life. As people trying to find a definite reason for this, it seems to be more than one anyway.


Say, using PHP, people encourage to add appropriate headers to make PDF stream opening on the fly directly in IE6 browser.



//size_of_stream is counted by bytes
//Display pdf text stream direct on the browser (IE bug fixed!)
//by setting the content-type


header("HTTP/1.1 200 OK");
header("Status: 200 OK");
header("Accept-Ranges: bytes");
header("Connection: Keep-Alive");


//Comment out for debugging purposes
//header("Cache-Control: public");
//Try setting 1 delta second for caching long enough for Adobe Addon to load PDF content
header("Cache-Control: public, max-age=1");
//Only need to specify User-Agent in Vary header as IE7 only accept that
//Default Vary header value is not welcomed at all
//Fixing IE7 bug for Vary header
header("Vary: User-Agent");
header('Pragma: public');
if (!is_null($size_of_stream)){header("Content-Length: ". $size_of_stream);}
header("Content-Type: application/pdf");
header('Content-Disposition: inline; filename="whatever.pdf"');
header("Content-Transfer-Encoding: binary\n");


This is almost true in many cases, including all the popular browsers like Firefox, Safari and Chrome. Yet there is still exception that PDF may not open properly on IE6, especially for small sized PDF stream.


Searching around for any server-side solution, it was really disappointed in terms of support of Internet Explorer 6.


Finally, I have a glimpse on a comment posted in the other forum which seems to work on almost any case with major version of Internet Explorer:


In Internet Explorer

  1. Select Tools
  2. Click Internet Options
  3. Select the Advanced tab
  4. Make sure "Do not save encrypted pages to disk" option near the bottom in the Security section is unchecked
This implies there is hardly any programming way to ensure PDF opening perfectly working.

Option "Do not save encrypted pages to disk" should only be enabled by default on Windows Server, whereas on most desktop PC the followings should be applied for security reason:


  1. Go to the Tools menu
  2. Click Internet Options
  3. Click the Advanced tab
  4. In the "Settings" box, scroll down to the section labeled "Security" 
  5. click to check the box next to the "Empty Temporary Internet Files folder when browser is closed" option
  6. Click OK to finish



This option does not delete cookies, but it will clear your cache of other files when you close your browser.

Now, the thing is how we are going to let people in an organization to follow this. This makes me unhappy:(




Related links:


http://joseph.randomnetworks.com/2004/10/01/making-ie-accept-file-downloads/#comment-670


http://robm.fastmail.fm/articles/iecachecontrol.html




However, this is not the end of the story.


To be enthusiastic for my work and diligent regarding all these, I keep digging deep into what's happening on a scenario which I have met in producing PDF on the fly.


For IE6, part of my PDF streams can be displayed correctly regardless of what option "Do not save encrypted pages to disk" disabled or not. When I tried to check the size of a successful PDF generated on the fly, they all show a bigger size like 120KB, 80KB or 76KB.


I did find some developer posts about the problem regarding PDF stream size to be displayed on IE. As I didn't remember wrong, someone mentioned about 8 Kilobytes in size to be minimum requirement on IE6.


When I go back to check that problematic PDF file by Firefox, it shows a size of 4 Kilobytes. Well, it's time to do an experiment on this.


Using PHP, it is easy to echo PDF stream first. Then you can calculate the size of stream by using function like strlen() to check size of string. Multiply this length by 1024, you'll get Kilobyte size yourself.


$len = strlen($pdf_stream)*1024; //length in bytes


Using simple algorithm to check it the length if it is less than 8192 bytes or not. When this is the case, you can first echo the PDF stream first.


echo $pdf_stream;


Then padding the spaces in advance:


for ($v=0;$v<=8192;$v++){
   echo ' '; //output space character
}


This makes sure the actual PDF output plus extra padding spaces occupying at least 8KB in size for IE compatibility.


Now, the problem is really solved across the browsers:)



Monday, May 16, 2011

Libgcc_s.so.1 bug on Linux

After a series of package installation onto Linux machine, some errors started to popup in the log. As found from a series of Google-search, it seems to be a problem widely spread among the software on Linux platform. This has been marked as a bug for attention.

Error message would be like this:
...libgcc_s.so.1: version 'GCC_4.2.0' not found (required by /usr/lib/libstdc++.so.6)...

This can happen on all kinds of software already installed onto the machine. People can barely find the cause until someone found a bug for this and suggest a fix - REMOVE IT!

However, it is safe to rename or move the problematic library to somewhere for retrieval. I would just rename it like this:

$ cd /destination_folder...
$ sudo mv libgcc_s.so.1 libgcc_s.so.1.bak

Sometimes you may want to check the file dependency before you take action:
$ ldd /destination_folder.../libgcc_s.so.1

The error should be gone so far.

Some useful materials can be found from the links below:
Fix "version `GCC_4.2.0' not found" for VMware 1.0.6 SErver and Ubuntu
`GCC_4.2.0' not found (required by /usr/lib/libstdc+

Wednesday, May 11, 2011

Base64 image fix for Internet Explorer

Although Internet Explorer 6 has been released for almost ten years, it is not likely that people are rushing to have a newer version, or even better alternative, of this piece of software among the hospitals.

For this famous version of Internet Explorer, trivial like unsupported image display in a way of data URI scheme seem to be annoying while nurses and doctors complain that they can't really see the chart images or even signature image (supposing in smaller size) on the web pages.

Although data uri images are widely supported by the other web browsers like Chrome, Firefox, Safari and Opera, there is strong reason for us to take care of IE. We have to accept that it is still a web browser which is widely spread among those ward computers, i.e., a niche market for web developer.

For any advancer who keeps using new stuff to catch the eyes, there will always be a dilemma on backward compatibility. By looking around in the sea of Google searches, I can barely find quite a lot of opinions on solving the problem. Most of them would urge you to keep focusing on the browsers. While some other developers suggest an intrusive way to modify almost every web page in the project, I prefer this way:


It was found from a blog article since 2005 but it sheds the light on a question:

Can we actually repair (fix) those images after they have been shown up on a incompatible browser?


The answer is yes. 


Here is the recipe:

  • JQuery core script file for integration.
  • A custom JQuery function in Javascipt file.
  • A min-sized PHP script file for Base64 image data processing.



By using client-side Javascript and an external PHP file, we can redirect in-line Base64 image stream data to external HTTP request for obtaining a compatible image object back from server-side image processing.

For nowadays web solution like PHP (v5.3) + JQuery (v1.6), it is not bad to review the source code from the above blog article and see what we can do for the new decade.


We would like to apply the fix with one function call fixBase64Image() once the web page is loaded completely. This function will search through the DOM elements and apply the fix to the target elements appropriately. A check on browser type and version is also possible to eliminate unnecessary action on non-related element. The target elements here are the image elements "img" with search criteria for the property of image source "src" which contains data uri stream like:



img src="data:image/gif;base64,..."



For client-side Javascript function (supposing JQuery has been involved in your project):

function fixBase64Image() {
 var BASE64_data = /^data:.*;base64/i;
 var BASE64_Path = "base64transfer.php";
 if ($.browser.msie){
  $("img").each(function(){
   // check matched image source
   if (BASE64_data.test($(this).attr("src"))) {
    // pass image stream data to external php
    var newSrc = BASE64_Path + "?" + ($(this).attr("src")).slice(5);
    $(this).attr("src",newSrc);
   }
  });
 }
 
};

The Javascript function will repair the broken images by replacing the source path of IMG element with an external PHP request to "base64transfer.php" while Base64 image data is encapsulated in HTTP request as the query string.


For the newly created external PHP file named "base64transfer.php",five lines of code are enough to support this:

$data = splitexplode(";", $_SERVER["QUERY_STRING"]);
$type = $data[0];
$data = splitexplode(",", $data[1]);
header("Content-type: ".$type);
echo base64_decode($data[1]);


This PHP script simply converts the query string, supposing a long string of Base64 image data, into raw image stream data and then send it back to the browser for image display.


This can be considered as a quick fix to the lack of support to Base64 in-line images in Internet Explorer 6.





Wednesday, March 2, 2011

MySQL Replication

There are good materials around for setting up multiple MySQL instance on local machine for data replication. But, the fact is that MySQL server only supports offline backup for INNODB type database. For online backup, Slave MySQL instance can be setup separately for backup purposes while Master instance can remain online to serve the users. Once Slave instance finishes its job, it can be startup again and synchronize with Master instance during the black-out period. This is a cost-effective solution for real-time MySQL backup.

The official manual have the details about this:
http://dev.mysql.com/doc/refman/5.5/en/replication.html

For basic tutorials, you may find it here:
http://forge.mysql.com/wiki/Replication/Tutorial

I love the presentation which bring a brief but clear information about what have to be done:
http://assets.en.oreilly.com/1/event/2/MySQL%20Replication%20Tutorial%20Presentation%202.pdf

To setup data replication on existing MySQL server, it is good to have a copy of /data directory first and then rename it to something like /data_slave and hook it up under the same parent folder of /data. This ensures a good start for setting up new database instance from the grounds up.

You may need to make a duplicate of my.ini which has already been used by the master MySQL instance. Rename that duplicate to something like, my2.ini and keep it in the same folder as the master my.ini file. Regarding the content of my2.ini, I suggest it is good to change the parameters first before starting those instances.

In my.ini, we have some basic settings like these:


[mysqld]
server-id=1
port            = 3306 
socket          = "C:/xampp/mysql/mysql.sock"
basedir="C:/xampp/mysql" 
tmpdir="C:/xampp/tmp" 
datadir="C:/xampp/mysql/data"
pid_file="mysql.pid"
innodb_data_home_dir = "C:/xampp/mysql/data"
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = "C:/xampp/mysql/data"
#innodb_log_arch_dir = "C:/xampp/mysql/data"

#Important setting for error prone builtin innodb plugin
innodb_use_sys_malloc = 0

innodb = ON

#Necessary for master instance
log-bin = mysql-bin

log_error="mysql_error.log"


In my2.ini, we should have changed those to:

[mysqld]
server-id=2
port            = 3307 
socket          = "C:/xampp/mysql/mysql2.sock"
basedir="C:/xampp/mysql" 
tmpdir="C:/xampp/tmp" 
datadir="C:/xampp/mysql/data2"
pid_file="mysql2.pid"
innodb_data_home_dir = "C:/xampp/mysql/data2"
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = "C:/xampp/mysql/data2"
#innodb_log_arch_dir = "C:/xampp/mysql/data2"

#Important setting for error prone builtin innodb plugin
innodb_use_sys_malloc = 0

innodb = ON

#Necessary for slave instance
relay-log = mysqld-relay-bin

log_error="mysql_error.log"

The concept behind this is that we are going to setup two MySQL instances on the same machine, with the master instance using port 3306 and the slave instance using port 3307, for communication. For your convenience, you may change it to any other port available on the server.

Supposing we have already have the master instance of MySQL started and running, now we need another one acting as a slave instance for backup purposes.

In Windows environment, it is necessary to setup a new Windows service running at the background for better system management.

Before this, you may want to test if my2.ini ready to be a running service. To test this in a debugging mode, it is good to use mysqld-debug.exe to start a debugging process with the target configuration file (my2.ini).

mysqld-debug.exe --defaults-file="c:\mysql\bin\my2.ini"

Take a look at the error log file (mysql_error.log) when anything goes wrong, you won't miss it.

When the test is finished and you are satisfied, you can create a new Windows service called mysql2 running in the background for easy management.

c:\mysql\bin\mysqld --install mysql2 --defaults-file="c:\mysql\bin\my2.ini"

After that, you may start this Windows service.

net start mysql2

It is recommended to use MySQL Workbench to manage those master and slave instance. For details, please read their manual.

You can download a copy of MySQL Workbench via: http://www.mysql.com/downloads/workbench/

It is also recommended to download the no-install.zip copy on Windows platform. The reason is that .msi installer file will require an pre-installation of .NET framework 3.5 or above which may install extra pieces of software like IIS web server while you might already have one instance of Apache server installed on the machine. They both race for port 80, anyway.

Once you can start both MySQL instances, you may start issuing commands in MySQL consoles to make data replication happens among them.

It's good to take a look at this first: http://dev.mysql.com/doc/refman/5.5/en/replication-howto-slaveinit.html

You may also check out the steps here: http://www.howtoforge.com/mysql_database_replication_p2

During this time, you may want to dump the data out of master instance and then feed it into slave instance for easy startup. Take a look at this: http://dev.mysql.com/doc/refman/5.5/en/replication-howto-masterstatus.html

To check the replication status on master instance, try the following commands in MySQL console:
mysql>unlock tables;


mysql> SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000020 |      107 |              |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)



mysql> SHOW SLAVE HOSTS;
+-----------+------+------+-----------+
| Server_id | Host | Port | Master_id |
+-----------+------+------+-----------+
|         2 |      | 3306 |         1 |
+-----------+------+------+-----------+
1 row in set (0.00 sec)


Once the replication happens, it will continue even after a system restart on the server itself. Sounds like a magic!

Monday, November 29, 2010

Build up a VMWare ESXi environment on a desktop PC

People have been discussing on the topics of setting up ESXi environment on a desktop PC either at home or for testing purposes. Although I have been searching on the forums to get some ideas why my installation has always been failed, the last conclusion I got is buying an Intel e1000 compatible network adapter first. Even the cheapest Intel network adapter will do a good job for you. Trust me, you won't need anything else but this. As I have ever seen, most forum users may get stuck in compiling not-quite-compatible onboard chipset NIC driver for use under ESX environment. This is not their fault, indeed. VMware makes Intel network card as a must-have component in the VMWare infrastructure.

I finally figured out that the most up-to-date ESXi server (say, version 4.1) can actually recognize the problematic Realtek onboard NIC, however, ONLY AFTER you insert an Intel e1000 network card into the slot. The thing is that the ESXi installation will only succeed once it detects Intel network adapter (obviously not the onboard NIC), making the network adapter as default management console's network interface to the outside world. Some people would just simply tell us having more Intel network adapters in stock for ESX maintenance and support.

After several trials with different versions of installation disc on my HP desktop PC, I found out another trap on the so-called HP customized version of ESXi Installable Disc which actually includes a timer program causing a crash even after the installation is completed. You may probably see a crash screen with error message once your ESXi server keeps running for the first 10 minutes. So you'd better off NOT using any customized version of ESXi Installable Disc. Use the official Installation Disc found on VMWare official site, please.

Regarding the memory, I can't get much but enough for 4GB RAM which is almost good for running a resources-hungry Windows server and a slim Linux based NAT router VM appliance like pfsense. Pfsense has a nice web GUI for you to do configuration via HTTP/HTTPS protocol. The use of NAT router is to do some port forwarding and maximize the use of one single public IP address to the WAN. You may probably use it as a portal to your VMs inside ESXi environment.

Once the installation is done with the aforementioned hardware added to your desktop PC, you can use another computer to connect it via VSphere client. Rememeber to use the IP address assigned to vmnic1 which is for management console connection only.



The architecture is simple enough. Assuming you have one onboard NIC vmnic0 and one Intel network adapter vmnic1, you are going to use vnmic1 for ESXi management console and vmic0 for the router.

Supposing you have created one virtual switch (vSwitch0) for management console connection by VSphere client and another virtual switch (vSwitch1) for VM pool including NAT router and other VMs, then you are good to go.

Here is the big picture