Friday, February 27, 2015

Reduce memory consumption on MySQL 5.6

It's been a while since I come back to check MySQL configuration on my development machine. Just feel a bit surprised that it looks like running really big databases within there. When I restarted it, I noticed that MySQL takes over 500 MB in memory usage. Actually, I didn't even start Apache server which comes with XAMPP package.

Current version of MySQL is 5.6.16. Even I'm quite happy with the database performance, it would be nice to save some resources for the other development tools running in parallel, especially for the big Eclipse IDE.

As long as it's not running as production machine, I'm fine with reducing the memory footprint a bit, actually, quite a bit.

Find my.cnf or my.ini for running MySQL instance and add/change the line as follow:

[mysqld]
performance_schema = off
Have a restart for MySQL service and the memory consumption is dropped back to under 100 MB which is fantastic!

Note: The MySQL Performance Schema is a feature for monitoring MySQL Server execution at a low level. You can turn it back on anytime for performance tuning by setting "performance_schema = on".





Tuesday, February 10, 2015

Installing PHP Accelerator ZendOpcahe on Apache 2.4 for Windows

It's Windows platform again and it's all about how to make it fast on running Apache server. It may be easier to find useful materials for Linux platform.

Supposing you are using XAMPP for Windows 1.8.x, you will feel curious about where ZendOpcache module is sitting. It's not there and you'll need to download it yourself.

Opcode caching is a good way to improve PHP performance as it takes out the compiling time of PHP source code for each consecutive loading after the first time that piece of PHP script is loaded and served.

Here're the steps:

1. Please download the package (version as of today is 7.0.4) via the following link:
http://windows.php.net/downloads/pecl/releases/opcache/7.0.4/php_opcache-7.0.4-5.4-ts-vc9-x86.zip

2. Stop Apache Service.

3. Download and extract the library file “php_opcache.dll” and copy it into the directory “C:\xampp\php\ext\”.

4. Add the following lines (marked in red) to the end of PHP configuration file:

.
.
.
C:\xampp\php\php.ini

 …

 ; ; set in php ini file only

 ; ; make sure it's readable (open_basedir is checked) by coverage viewer script

 ; xcache.coveragedump_directory = ""

 ; ;; END OF XCACHE CONFIG



 [Zend_Opcache]

 zend_extension = "C:\xampp\php\ext\php_opcache.dll"

 opcache.memory_consumption=128

 opcache.interned_strings_buffer=8

 opcache.max_accelerated_files=4000

 opcache.revalidate_freq=60

 opcache.fast_shutdown=1

 opcache.enable_cli=1

.
.
.

5. Start Apache service.

6. Examine and check if the libraries are loading properly via http://localhost/xampp/phpinfo.php


Configuration of mod_fcgid in XAMPP Apache for Windows

First thing first, which one is the fastest of serving PHP source?

Here's a nice comparison on various PHP implementations in Apache:

http://2bits.com/articles/apache-fcgid-acceptable-performance-and-better-resource-utilization.html

Interesting points here:


Apache's mod_php
Apache's mod_php is the most widely used mode for PHP with Apache. mod_php itself is the entire PHP interpreter embedded in each Apache process that gets spawned. This provides performance and stability benefits, e.g.
No need to call an external process (e.g. CGI).
No need to communicate with another process via sockets (e.g. Fast CGI).
The APC cache is shared by all Apache processes.
It also has some disadvantages
The memory footprint per Apache process is large, specially when sites indulge in contributed modules.
If Apache is serving static content, e.g. images and CSS files, it still has to spawn large processes because of the embedded PHP interpreter.


CGI
CGI (Common Gateway Interface) is the legacy way of runing applications on the web from the mid 1990s or so. It was too inefficient for anything but small sites. CGI spawns a new process for every incoming request to execute a PHP script, a very resource intensive and inefficient way of doing things. No wonder it faded away over time as web applications became more complex.


FastCGI
FastCGI was introduced to avoid some of the issues with running languages, including PHP, inside the Apache process, as well as avoiding the inefficiency of CGI.
A FastCGI application is executed outside of the web server (Apache or other wise), and waits for requests from the web server using a socket. The web server and the FastCGI application can even be on separate physical machines and communicate over the network.
Because the web server and the application processes are separate better isolation is possible.
In reality, running PHP as mod_fastcgi with Apache has proved to be problematic. Mainly with stability. Even on Drupal.org we tried it for a while, but switched back to mod_php after some time.


mod_fcgid
mod_fcgid was introduced to be binary compatible with FastCGI, but with better control over spawning processes. The benefits of process isolation are still there.


Looks like a brief history on how CGI evolves.

mod_fcgid is a high performance alternative to mod_cgi or mod_cgid, which starts a sufficient number instances of the CGI program to handle concurrent requests, and these programs remain running to handle further incoming requests.

The Apache Software Foundation and the Apache HTTP Server Project are pleased to announce the release of version 2.3.9 of mod_fcgid, a FastCGI implementation for Apache HTTP Server versions 2.0, 2.2, and 2.4. This version of mod_fcgid is a security release.

I have been searching around for the way to implement mod_fcgid in Apache 2.4 for Windows which is embedded in XAMPP 1.8.x package.

Here's the procedure that would work on Windows:

1. Please download the zip package via the following link:
https://www.apachelounge.com/download/win32/modules-2.4/mod_fcgid-2.3.9-win32.zip

2. Stop Apache service.

3. Expand and extract the library file “mod_fcgid.so” and then copy it under the directory C:\xampp\apache\modules\.

4. Create new configuration file with the following content (adjust the actual paths of XAMPP installation directory to suit the needs):

C:\xampp\apache\conf\extra\httpd-fcgid.conf

##################################################################

 # Configuration for mod_fcgid

 # FileName: conf/extras/mod_fcgid_php.conf







 # Fast CGI module Settings (PHP 5.3, 5.4)



  FcgidInitialEnv PHPRC "C:\\xampp\\php"

  FcgidInitialEnv PATH "C:\\xampp\\php;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;"

  FcgidInitialEnv SystemRoot "C:\\Windows"

  FcgidInitialEnv SystemDrive "C:"

  FcgidInitialEnv TEMP "C:\\WINDOWS\\TEMP"

  FcgidInitialEnv TMP "C:\\WINDOWS\\TEMP"

  FcgidInitialEnv windir "C:\\WINDOWS"



 # Fast CGI module Settings (PHP 5.2)



 # FcgidInitialEnv PHPRC "C:/php/"

 # FcgidInitialEnv PATH "C:/php;C:/WINDOWS/system32;C:/WINDOWS;C:/WINDOWS/System32/Wbem;"

 # FcgidInitialEnv SystemRoot "C:/Windows"

 # FcgidInitialEnv SystemDrive "C:"

 # FcgidInitialEnv TEMP "C:/WINDOWS/TEMP"

 # FcgidInitialEnv TMP "C:/WINDOWS/TEMP"

 # FcgidInitialEnv windir "C:/WINDOWS"



 # Global mod_fcgid settings



  FcgidIOTimeout 40

  FcgidConnectTimeout 10

  FcgidMaxProcesses 8

  FcgidOutputBufferSize 64

  ProcessLifeTime 240

  FcgidMaxRequestsPerProcess 500

  FcgidMinProcessesPerClass 0





 # Global Config Example

 # Comment out next 4 lines to use per-Directory or per-VirtualHost configuration

  

     #Options ExecCGI

  Options Indexes FollowSymLinks ExecCGI

     AddHandler fcgid-script .php

     FcgidWrapper "C:/xampp/php/php-cgi.exe" .php

  



 #

 # Examples below show how you can use per-Directory and per- VirtualHost Configs

 # You can use multiple Directory and VirtualHost Configs at the same time

 #



 # Per Directory Config Example

 #

 #

 #  FcgidInitialEnv PHPRC "C:/php"

 #  AddHandler fcgid-script .php

 #  Options Indexes FollowSymLinks ExecCGI

 #  AllowOverride all

 #  FcgidWrapper "C:/php/php-cgi.exe" .php

 #  Require all granted

 #





 # Per VirtualHost Config Example

 #

 #

 # DocumentRoot  /Apache22/htdocs/fcgi

 # ServerName fcgi.local

 # ErrorLog logs/fcgi.error.log

 # CustomLog logs/fcgi.access.log common

 # FcgidInitialEnv PHPRC "C:/php"

 #

 #  AddHandler fcgid-script .php

 #  Options Indexes FollowSymLinks ExecCGI

 #  AllowOverride all

 #  FcgidWrapper "C:/php/php-cgi.exe" .php

 #   Require all granted

 #

 #





 # Do not comment out below line



 ######################################################################





5. Add new lines to the following configuration file:
C:\xampp\apache\conf\httpd.conf
…

#LoadModule watchdog_module modules/mod_watchdog.so

#LoadModule xml2enc_module modules/mod_xml2enc.so

LoadModule fcgid_module modules/mod_fcgid.so





…

#FCGI Module

 Include "conf/extra/httpd-fcgid.conf"

#End of file





6. Comment out several lines to the following configuration file:
C:\xampp\apache\conf\extra\httpd-xampp.conf
…

#
# PHP-Module setup
#
LoadFile "C:/xampp/php/php5ts.dll"
LoadFile "C:/xampp/php/libpq.dll"
LoadModule php5_module "C:/xampp/php/php5apache2_4.dll"

#Comment out to use purely mod_fcgid to serve all .php files, see httpd-fcgid.conf
#
#    SetHandler application/x-httpd-php
#
#
#    SetHandler application/x-httpd-php-source
#
7. Start Apache service and test the web pages.

8. Examine and check if the libraries (including mod_fcgid) are loading properly via http://localhost/xampp/phpinfo.php







Saturday, January 31, 2015

Pfsense 2.2 Upgrade and OpenVM Tools installation

Pfsense 2.2 is out now whereas I couldn't wait to upgrade the guest VM for this.

Then OpenVM Tools seems to be unable to load up. 

Errors appear in the login console:
KLD vmmemctl.ko: depends on kernel - not available or version mismatch
linker_load_file: Unsupported file type
KLD vmxnet.ko: depends on kernel - not available or version mismatch
linker_load_file: Unsupported file type
KLD vmblock.ko: depends on kernel - not available or version mismatch
linker_load_file: Unsupported file type
KLD vmhgfs.ko: depends on kernel - not available or version mismatch
linker_load_file: Unsupported file type

Using command kldstat only shows up somthing like this:
$ kldstat
Id Refs Address    Size     Name
 1    8 0xc0400000 93ca60   kernel
It sounds like vmxnet.ko and the other OpenVM drivers not loading up properly.

Pfsense 2.2 is based on FreeBSD 10.1 which may not be compatible with the way that works in FreeBSD 9.x. Someone in the forum suggested that:

For now Vmware is not supporting FreeBSD version 10. You must wait with upgrade to VSphere 6.

Yet we can't wait, can we? 

Quick solution is trying to install OpenVM Tools from FreeBSD ports via pfsense console itself:

pkg install open-vm-tools-nox11

After installation, try the following as well:

To run the Open Virtual Machine tools at startup, add the following
settings to your /etc/rc.conf :
vmware_guest_vmblock_enable="YES"
vmware_guest_vmhgfs_enable="YES"
vmware_guest_vmmemctl_enable="YES"
vmware_guest_vmxnet_enable="YES"
vmware_guestd_enable="YES"
After a reboot, vmxnet.ko drivers should load up again as follows.
$ kldstat
Id Refs Address    Size     Name
 1    8 0xc0400000 93ca60   kernel
 2    1 0xc0d3d000 6a32c    acpi.ko
 3    1 0xc20d1000 3000     vmmemctl.ko
 4    1 0xc20f5000 4000     vmxnet.ko
 5    1 0xc20f9000 5000     vmblock.ko
 6    1 0xc20fe000 a000     vmhgfs.ko






Thursday, November 20, 2014

Realtime & Sacalable RDBMS on Hadoop

Big Data has been a hot topic among business and technology companies while they are actively embracing new Hadoop based technologies like HiveQL and HDFS for business decision making. This also implies a new trend in both hardware infrastructure and software development. Although Big Data might not be applicable to all aspects which traditional RDBMS has been leading, there is a growing demand that drives the service providers to think about how to quickly absorb customers with existing infrastructure based on RDBMS.

A new kind of Hadoop database was born to meet the requirement for scaling out the petabyte level RDBMS database within Hadoop infrastructure. It keeps the features of RDBMS while it can scale out like NoSQL. The product called Splice Machine has its major version 1.0 released as the first ever Hadoop RDBMS in the market. This implies that the code change can be greatly minimised in order to move existing SQL based apps into Hadoop infrastructure. This 2 year old company is started up by CEO cofounder Monte Sweben via fundraising US$19 million from Mohr Davidow Ventures, InterWest Partners. It claims to be a real-time transactional SQL on Hadoop database and a direct competitor to the SQL giant Oracle.



Trial version can be downloaded via filling out the form at http://www.splicemachine.com/product/download/.

The download link will be emailed to your inbox with an expiry time of 2 hours.

Splice Machine can be installed in two modes:

Standalone: A commodity machine should have at least 8GB of RAM, with 4+ GB available,
and at least 3x the available disk as data you intend to load. It can be installed on Mac/Win+Cygwin/Linux.

Cluster: Each commodity machine should have at least 15GB RAM and at least 3x the available
disk as data you intend to load. It can only be installed on Linux with platform like Cloudera CDH 4.x/5.x, Hortonworks HDP 2.x and MapR 4.x.

Version 1.0 brings a number of new features.  For details, please read http://doc.splicemachine.com
  • Native Backup and Recovery
  • User/Role Authentication and Authorization
  • Parallel, Bulk Export
  • Data Upsert
  • Management Console for Explain Trace
  • MapReduce Integration
  • HCatalog Integration
  • Analytic Window Functions
  • Log Capture Capability












Monday, November 3, 2014

Technical Support Levels that make sense

Think about the situation on how the people work together in the team and everybody takes on one's role. To work effectively, they divide into small teams to take care of the problems in particular fields or levels. This helps forwarding the problem to the target specialists.

For I.T. service management, typical technical support are generally divided into 3 to 4 different levels:

Tier 1 support - Incident Management:

They are the frontline specialist customers meet over the counter everyday. Their abilities range from assistance in handling simple problems or generic 'how to' questions. All these queries are addressed by a Tier 1 specialist. Most Tier 1 issues are generic or FAQs, which are answered either through the Service Knowledge Management System (SKMS) or in other forms to the support executive. They also need to update those information or FAQs within the self service view on the service portal. They solve about 80 percent of user problems, including such issues as:


  • Problems with usernames and passwords
  • Physical layer issues
  • Verification of hardware and software setup
  • Installation, reinstallation and uninstallation issues
  • Menu navigation

Tier 2 support - Problem & Change Management:

Tier 2 comes into play when the Tier 1 technician is unable to solve the query within the set resolution time frames. This escalation may arise out of the product/device, which may be technically complex therefore requiring the intervention at Tier 2.
The nature of Tier 2 queries may range from advanced features, product bugs or failures. The task here is to diagnose and resolve issues related to these applications and components by:
  • Understanding the environment in which the customer operates and funnel it to the specific problems.
  • Undertaking simulation checks in a lab, if the problem remains untraceable.

Tier 3 support - Engineering Support:

Tier-3 support acts as a support team on behalf of the technology creator. The aim is to find a solution from the technology creator as the nature of problem is complex and needs design level interaction like developing a new patch and release it. 
These specialists handles the most difficult problems and are experts in their field, sometimes assisting both level 1 and level 2 specialists. They also research and develop solutions for new or unknown issues.

Tier 4 support -Vendor Support (optional):

The optional fourth level of support is sometimes provided by either a software or hardware vendor and their management team on special issues.







FHIR in FIVE minutes


HL7 V.2 has been so popular for years in Australia while HL7 V.3 has been established and adopted in U.S.. However, there are lots of comments and opinions about how it is difficult to implement system in V.3 as it sounds like an XML edition of HL7 protocol. Particular reason for the unpopularity of V.3 may be laying on whether it is worthy to migrate existing V.2 assets to V.3 format while existing systems are still supporting V.2 standard.

If you want to know more about how HL7 V.2 evolved to V.3, please read the PDF document via the link.

Let's take a look at the timeline of the development of HL7 standards:


Things are moving on and so does this standard. In recent years, new toolsets have been released and named as FHIR (pronounced "Fire"). Fast Healthcare Interoperability Resources (FHIR) defines a set of "Resources" that represent granular clinical concepts.

XML and then FHIR. Sounds complicated, isn't it? What's this set of "Resources" all about?

HL7 V.3 is about XML formatted messaging while FHIR is about the way to deliver it.

As long as we learn XML as the key message format of data exchange between the systems, we may not miss the term Web Service which actually encapsulates XML message with an envelope and then send it out to the other side of the receiving end.

One category of Web Service is using REST-ful (REpresentational State Transfer) API for communication. This type of Web Service makes data communication easier as it includes basic CRUD (Create, Read, Update, Delete) operations over HTTP/HTTPS protocol (standard web technologies). You can even open a browser and read the results in XML format via the pre-defined URL. After all, it's producing human readable messages like HL7 V.3.

For the feature like RESTful API, System Developers can surely create lots of possibility, especially for Mobile Apps (IOS, Android, etc.). System Integrators can bridge up the systems through the orchestration and collaboration of Web Services. All these can be summarised into a typical usage of Service Oriented Architecture (SOA).

In other words, through FHIR it is possible to produce a set of reusable resources in the form of URLs like:
http://pat.registry.org/Patient/2231
http://hospitalA.org/Practitioner/870
http://hospitalA.org/Organization/10
http://lab.hospitalA.org/Observation/3ff12
http://lab.hospitalA.org/DiagRep/4545

Common resource's identity would be like this:


The business logic on top of FHIR is the coding (Java, .NET, C++) to collaborate these web services in order to achieve the desired outcome. FHIR server is basically one kind of web servers but includes the engine to support RESTful operations for clinical data.



FHIR is still under active review and development so you may want to catch up with their progress via the URLs:

Current specification:
http://www.HL7.org/fhir

Repository of packages and tutorials:
http://gforge.hl7.org/gf/project/fhir/scmsvn/?action=browse&path=%2Ftrunk%2F

Demo (Grahame's test server):
https://fhir.healthintersections.com.au












Friday, October 31, 2014

Business Change Management as a glance

Change, as we know it, is something happening in our daily life while everybody needs to adapt to it. But a change in the business process or workflow would not only affect one, but also a team of people around you. What are you going to deal with the change as a process owner? And what do you expect the team to do in coping with the change? These are serious questions as they end up affecting the business ultimately. People are serious when talking about Business anyway.

Warm-up questions before you think in depth about a big business change:
What's Change Management?
Have you heard about ADKAR?
How would you incorporate Change Management in ITS Development Life Cycle?

Business Change Management focuses on how people deal with a change. It's more than a single event or activity. BCM is the actual process to anticipate and analyze those impacts of a project from the USER's perspective. It guides and informs project activities and planning to help manage user's EXPECTATIONs, RESISTANCE and buy-in. The projects are more likely to success with a clearly defined Change Management TEAM and APPROACH.

Change Management Team provides:

  • Readiness assessments
  • Communications
  • Training
  • Support planning after the change
  • Stakeholder review through post-deployment
In a sense of ITIL service management, CM can be interpreted in a way of Continuous Service Improvement (CSI).

Around a change, there are 3 key aspects to be involved:
  • People: Stakeholders, business owner, service owner, operational team
  • Process: Business processes involved in the change
  • Tools: Knowledge and technology


The flow of CSI Model in change management could be like this:

  1. Evaluate
  2. Assess
  3. Design
  4. Implement
  5. Manage Change (When finished, the cycle will loop to the phase 1 - Evaluate again.)

Apart from that, there are researches out there studying about the implications of the flow of CM.

One of the famous methodologies is ADKAR Model (founded by Prosci Research). Using ADKAR Model, BCM Team can MANAGE and MONITOR fundamental elements of a change caused by a new system and project.

The ADKAR Model involves the 5 phases:
  1. Awareness: Understand NEEDS & NATURES of the change
  2. Desire: SUPPORT the change by participation and engagement activities
  3. Knowledge: LEARN how to change and new skills & behaviours
  4. Ability: IMPLEMENT the change and DEMONSTRATE performance
  5. Reinforcement: SUSTAIN the change & BUILD a CULTURE around the change







Wednesday, October 29, 2014

A bit of Agile Software Development

As the developers are always asked which methodology would be adopted in their software development life cycle. Most of the time they would say, "In a Agile way!" So, how actually should Agile development look like in real life?

Consider doing software development in Agile way, there are 12 principles which I can easily forget.

Now, let's forget them one-by-one:

The Agile Manifesto is based on 12 principles:

  1. Customer satisfaction - Make it QUICK and CLEAN in terms of software delivery.
  2. Welcome changing requirements - Be YES-MAN to your client, as always.
  3. Frequent software delivery - Release it by weeks/days, of course, with QA tests done.
  4. Close, daily cooperation - Love your business and talk to the client everyday.
  5. Projects built around motivated & trusted individuals - Yes, we trust them so we work with them.
  6. Face-to-face conversation - It means you should be there in front of your parties for TALKING
  7. Principal measure of progress - It basically means NO Progress if software is NOT YET WORKING. So try your best.
  8. Sustainable development - Keep your dev work RUNNING in constant pace.
  9. Continuous attention to technical excellence and good design - We loves good things like STATE-OF-THE-ART/BLEEDING-EDGE technology.
  10. Simplicity - LEAVE traces to ongoing work as much as you can, i.e., We don't have time for sure. Just keep it simple & get the work done.
  11. Self-organizing teams - CONTROL YOURSELF.
  12. Regular adaptation to changing circumstances - BE ALERT to the changes around you, and the project.

Tuesday, October 28, 2014

The A Team for Web Portal Consultancy

Can you imagine how difficult for you to convince the client that you have got the strong team to support their big tasks in constructing a web portal?

Here comes the list of roles to be involved in Web Portal Consultancy team:

Digital Strategist:

Digital strategy was the discipline of working with teams inside the agency and/or directly with the brand to solve complex business and marketing problems.
Digital strategists are the people that lead the problem solving charge and help connect the dots between business, brand, and marketing goals and the channels, tactics and technologies that’ll make it all come together to provide actionable results. Competent digital strategists work in a highly focused manner with the client and/or agency business unit to get a clear and detailed understanding of what the challenges are from a business point-of-view [like a business analyst would]. Of course, at any given time during the engagement, you can swap out “business” with “creative”, “technology”, etc. Same strategic approach, different lens.

Web Technical Architect:

The Technical Architect is responsible for the overall technical design and build of the custom elements of the solution. The Technical Architect works as a team member along with the Engagement Manager, Developer and Solutions Architect to deliver the complete solution for the customer. This role must be organized and analytical, adept at working in a team environment, able to design and implement a project schedule, and able to handle multiple priorities.

Web Information Architect:

Information architects organize the content of web sites, intranets and online communities in a user-friendly way that allows visitors to quickly find what they're searching for. They then create interfaces to support that organization. Information architects begin by analyzing the target audience and level of interactivity, and technology required, in addition to the data presented through the site. They then develop a plan that will balance efficiency with ease-of-use. Information architects work with graphic and web designers, database engineers and coders to implement their plans.

Web Business Analyst:

Web Business Analyst will act as primary, working with clients and agency partners in helping plan, document and consult on new projects. In addition, the Web Business Analyst will act as a technical resource assisting the client team with presales and estimates, along with serving as the gateway to the development team; interfacing with production, operations and Sr. Management.

Web Designer:

Web designers plan, create and code web pages, using both non-technical and technical skills to produce websites that fit the customer's requirements. They are involved in the technical and graphical aspects of pages, producing not just the look of the website but determining how it works as well. Web designers might also be responsible for the maintenance of an existing site.

Web Developer:

Similar to Web designer, Web developer is a more specialist role, focusing on the back-end development of a website and will incorporate, among other things, the creation of highly complex search functions.

Web Tester:

Web Tester help testing web application user interface and mobile applications. User Interface testing includes verification against Visual Design specifications and the usability assessment. The tester has to assure that the web application UI conforms to the requirements, and report a defect when it doesn't.

Web Design Quality Assurance:

Web Design Quality Assurance ensure the website adheres to the Web Design Guidelines like HTML standards, CSS standards, W3C standards, Accessibility standards, Performance and Cross Browser Compatability.

Web Development Quality Assurance:

Web Development and QA Specialist ensures the web assets are operationally sound and perform in accordance with the organization’s technical standards, including reviewing and analyzing the site, and all the systems that provide a public interface, for issues of quality and infrastructure performance. They are responsible for the design, development and implementation of new system functionality for the website as required and the design of quality assurance procedures for all stages of the system process change, and coordinating a change control process for implementing technical and other updates in a timely and non-disruptive manner.