Realtime & Sacalable RDBMS on Hadoop

Big Data has been a hot topic among business and technology companies while they are actively embracing new Hadoop based technologies like HiveQL and HDFS for business decision making. This also implies a new trend in both hardware infrastructure and software development. Although Big Data might not be applicable to all aspects which traditional RDBMS has been leading, there is a growing demand that drives the service providers to think about how to quickly absorb customers with existing infrastructure based on RDBMS.

A new kind of Hadoop database was born to meet the requirement for scaling out the petabyte level RDBMS database within Hadoop infrastructure. It keeps the features of RDBMS while it can scale out like NoSQL. The product called Splice Machine has its major version 1.0 released as the first ever Hadoop RDBMS in the market. This implies that the code change can be greatly minimised in order to move existing SQL based apps into Hadoop infrastructure. This 2 year old company is started up by CEO cofounder Monte Sweben via fundraising US$19 million from Mohr Davidow Ventures, InterWest Partners. It claims to be a real-time transactional SQL on Hadoop database and a direct competitor to the SQL giant Oracle.

Trial version can be downloaded via filling out the form at http://www.splicemachine.com/product/download/.

The download link will be emailed to your inbox with an expiry time of 2 hours.

Splice Machine can be installed in two modes:

Standalone: A commodity machine should have at least 8GB of RAM, with 4+ GB available,

and at least 3x the available disk as data you intend to load. It can be installed on Mac/Win+Cygwin/Linux.

Cluster: Each commodity machine should have at least 15GB RAM and at least 3x the available

disk as data you intend to load. It can only be installed on Linux with platform like Cloudera CDH 4.x/5.x, Hortonworks HDP 2.x and MapR 4.x.

Version 1.0 brings a number of new features.  For details, please read http://doc.splicemachine.com

• Native Backup and Recovery
• User/Role Authentication and Authorization
• Parallel, Bulk Export
• Data Upsert
• Management Console for Explain Trace
• MapReduce Integration
• HCatalog Integration
• Analytic Window Functions
• Log Capture Capability

Technical Support Levels that make sense

Think about the situation on how the people work together in the team and everybody takes on one's role. To work effectively, they divide into small teams to take care of the problems in particular fields or levels. This helps forwarding the problem to the target specialists.

For I.T. service management, typical technical support are generally divided into 3 to 4 different levels:

Tier 1 support - Incident Management:

They are the frontline specialist customers meet over the counter everyday. Their abilities range from assistance in handling simple problems or generic 'how to' questions. All these queries are addressed by a Tier 1 specialist. Most Tier 1 issues are generic or FAQs, which are answered either through the Service Knowledge Management System (SKMS) or in other forms to the support executive. They also need to update those information or FAQs within the self service view on the service portal. They solve about 80 percent of user problems, including such issues as:

• Problems with usernames and passwords
• Physical layer issues
• Verification of hardware and software setup
• Installation, reinstallation and uninstallation issues
• Menu navigation

Tier 2 support - Problem & Change Management:

Tier 2 comes into play when the Tier 1 technician is unable to solve the query within the set resolution time frames. This escalation may arise out of the product/device, which may be technically complex therefore requiring the intervention at Tier 2.

The nature of Tier 2 queries may range from advanced features, product bugs or failures. The task here is to diagnose and resolve issues related to these applications and components by:

• Understanding the environment in which the customer operates and funnel it to the specific problems.
• Undertaking simulation checks in a lab, if the problem remains untraceable.

Tier 3 support - Engineering Support:

Tier-3 support acts as a support team on behalf of the technology creator. The aim is to find a solution from the technology creator as the nature of problem is complex and needs design level interaction like developing a new patch and release it. 

These specialists handles the most difficult problems and are experts in their field, sometimes assisting both level 1 and level 2 specialists. They also research and develop solutions for new or unknown issues.

Tier 4 support -Vendor Support (optional):

The optional fourth level of support is sometimes provided by either a software or hardware vendor and their management team on special issues.

FHIR in FIVE minutes

HL7 V.2 has been so popular for years in Australia while HL7 V.3 has been established and adopted in U.S.. However, there are lots of comments and opinions about how it is difficult to implement system in V.3 as it sounds like an XML edition of HL7 protocol. Particular reason for the unpopularity of V.3 may be laying on whether it is worthy to migrate existing V.2 assets to V.3 format while existing systems are still supporting V.2 standard.

If you want to know more about how HL7 V.2 evolved to V.3, please read the PDF document via the link.

Let's take a look at the timeline of the development of HL7 standards:

Things are moving on and so does this standard. In recent years, new toolsets have been released and named as FHIR (pronounced "Fire"). Fast Healthcare Interoperability Resources (FHIR) defines a set of "Resources" that represent granular clinical concepts.

XML and then FHIR. Sounds complicated, isn't it? What's this set of "Resources" all about?

HL7 V.3 is about XML formatted messaging while FHIR is about the way to deliver it.

As long as we learn XML as the key message format of data exchange between the systems, we may not miss the term Web Service which actually encapsulates XML message with an envelope and then send it out to the other side of the receiving end.

One category of Web Service is using REST-ful (REpresentational State Transfer) API for communication. This type of Web Service makes data communication easier as it includes basic CRUD (Create, Read, Update, Delete) operations over HTTP/HTTPS protocol (standard web technologies). You can even open a browser and read the results in XML format via the pre-defined URL. After all, it's producing human readable messages like HL7 V.3.

For the feature like RESTful API, System Developers can surely create lots of possibility, especially for Mobile Apps (IOS, Android, etc.). System Integrators can bridge up the systems through the orchestration and collaboration of Web Services. All these can be summarised into a typical usage of Service Oriented Architecture (SOA).

In other words, through FHIR it is possible to produce a set of reusable resources in the form of URLs like:
http://pat.registry.org/Patient/2231
http://hospitalA.org/Practitioner/870
http://hospitalA.org/Organization/10
http://lab.hospitalA.org/Observation/3ff12
http://lab.hospitalA.org/DiagRep/4545

Common resource's identity would be like this:

The business logic on top of FHIR is the coding (Java, .NET, C++) to collaborate these web services in order to achieve the desired outcome. FHIR server is basically one kind of web servers but includes the engine to support RESTful operations for clinical data.

FHIR is still under active review and development so you may want to catch up with their progress via the URLs:

Current specification:
http://www.HL7.org/fhir

Repository of packages and tutorials:
http://gforge.hl7.org/gf/project/fhir/scmsvn/?action=browse&path=%2Ftrunk%2F

Demo (Grahame's test server):
https://fhir.healthintersections.com.au