This past winter, Red Hat announced the release of a product called MRG–a computing platform that features high-speed messaging and allows high-throughput computing, realtime transactions, and workload management. Not sure what all that means? We weren’t either. So we contacted Brian Che, the project manager for MRG, to see if we couldn’t get a few questions answered. He obliged, and so we bring you the MRG QandA. Still have questions of your own you want answered? Comment and let us know…

Red Hat has been working on the technologies behind MRG for quite some time–each of the components in MRG has had years of development. For example, Red Hat has been working on realtime technologies in the upstream kernel community for over seven years. Messaging has had a
similarly lengthy development history. Condor, the technology behind our grid scheduler, started development in the 1980’s!

We started work on these technologies because we saw the need for these capabilities, even if we didn’t know when or how we were going to bring
these technologies to market yet. For example, messaging is at the heart of enterprise computing. We had needs for messaging infrastructure at Red Hat–for building out our own capabilities around things like virtualization management. Many of Red Hat’s customers were asking us to provide an open source messaging offering. So, we started working on the AMQP specification and our messaging implementation, even though we didn’t know it was going to end up in something called “Red Hat Enterprise MRG”.

As we started working with customers and the community around the various technologies in MRG, it became apparent to us that the technologies had reached a point of maturity where we could support our most demanding customers with them. Also, we saw significant opportunities for building out fundamentally new capabilities by integrating messaging, realtime, and grid into one platform. And so, MRG was born.

We released MRG v1 at the Red Hat Summit on June 19, 2008. MRG v1 offers support for messaging and realtime, and grid is in Technology Preview. We’ll release a 1.1 update to MRG that will bring grid into full support as well.

JP Morgan Chase, like other investment banks, uses messaging for everything from executing stock trades to providing feeds of market data
to internal data distribution. They currently send over 1 billion AMQP messages per day.

Realtime provides deterministic performance. The US Navy is deploying realtime in its DDG 1000 naval destroyers. Realtime is critical in this
environment, because the ships’ computers have to respond precisely without ever pausing, freezing, or getting out of sync with other
events. Otherwise, the results could be disastrous.

One of our large manufacturing customers has been working with Red Hat to build an on-demand grid in Amazon’s EC2 cloud environment for the times it needs access to a grid for calculations. Because this customer isn’t able to utilize fully a dedicated grid, having the option to deploy a grid in the cloud provides them significant cost savings and flexibility.

There isn’t an ideal customer–ultimately, we think that almost any customer will benefit from MRG. MRG provides a new platform and solution for many of the most pressing problems that enterprises face today. We have significant customer interest from many industries.

Having said that, many of our largest customers are MRG early adopters, such as investment banks like JP Morgan Chase, telco companies like
Alcatel Lucent, and multiple agencies in the US Government. We are also working across oil&gas, animation studios, Internet, shipping, stock exchanges, defense, travel, and so on.

MRG takes special advantage of and is highly optimized for Linux to deliver its performance. Additionally, at Red Hat, we have been driving
changes into Linux itself in order to benefit things like messaging performance. So, the fact that we are focusing on just one platform and optimizing both that platform and our implementation on that platform gives us tremendous gains (Note: everything we do is open source and contributed back to the community).

For example, we have written a new high performance journal for durable or persistent messaging that is highly tailored to Linux’s I/O model.
By using this journal, MRG Messaging can achieve throughputs up to about 500,000 durable messages/second/LUN. This rate is about 100 times
faster than other messaging solutions. For more details, you can read Carl Trieloff’s entry in the Red Hat Press blog.

Yes. For example, we support messaging clients across a wide variety of platforms and languages, from Linux to Solaris to Windows, and from C++ to Java/JMS to scription languages like Python. On the grid side, we’ll support scheduling to both Linux and Windows. And, of course, since we integrate with virtualization, this gives us a lot of flexibility in running on other operating systems.

We definitely see a lot of interest in cloud computing from customers. MRG integrates with cloud providers like Amazon EC2 so that you can dynamically provision and add capacity in the cloud from your grid scheduler. This means, for example, that you could have a scenario where you fully utilize your local data center but have additional work you want to compute.

MRG can automatically provision, say, 1000 extra servers for you at EC2, send your work over, get your results back, and tear down the servers when you’re done–all automatically. Some of our other customers are looking at provisioning most or all of their capacity in the cloud because they won’t utilize a data center fully and want to save on capital expenses.

In either case, one of the powerful features of MRG is that it can blend local capacity with cloud capacity. This means you don’t get locked into one cloud provider, and you can grow your infrastructure dynamically in the cloud or in your local data center.

One of the significant things about AMQP is that it is the first protocol standard for business messaging. All other standards, like JMS, aren’t comprehensive enough and don’t specify down to the wire level to provide true interoperability and an open ecosystem. So, I’m not concerned about competing standards–there aren’t really any right now. That’s why there is so much interest in AMQP.

I think that most big businesses will understand and appreciate what AMQP has to offer. Notably, many of the big businesses driving AMQP are not vendors but users. Eventually, if you want to work with these users, you’re going to have to adopt AMQP.

MRG is important to all of our offerings–it’s pretty strategic and central to many of the things that Red Hat is doing. MRG adds realtime capabilities to Red Hat Enterprise Linux and enables you to provide flexible scalability and performance for applications running on Red Hat Enterprise Linux. We’re working with Red Hat Network so that you can provision and manage MRG with our standard management tools. MRG Realtime and a realtime JVM from IBM or Sun can provide deterministic performance for JBoss Java applications. We’re working with the JBoss team to support MRG Messaging as a messaging transport for the JBoss ESB. And, many of our core products and technologies are using MRG technology. IPA and oVirt, for example, are both leveraging our messaging capabilities for distributing data.

More information

• Read more about MRG at the Red Hat Press blog.
• See the official MRG announcement.
• The official MRG pages on redhat.com.

Following up on my Previous Post I decided to do little test to see how accurate stats we can get for for Index Stats created by ANALYZE TABLE for MyISAM and Innodb.

But before we go into that I wanted to highlight about using ANALYZE TABLE in production as some people seems to be thinking I advice to use it.... a lot. In fact I should say I see more systems which have ANALYZE abused - run too frequently without much need than systems which do not run ANALYZE frequently enough.

First it is worth to note MySQL only saves very basic cardinality information for index prefixes for index stats and these rarely change. There is no histograms or any other skew metrics etc. MySQL optimizer also uses number of rows in the table for many decisions but this is computed live (maintained for MyISAM and estimated during query execution for Innodb). This basic information means it does not change whole that quickly at extent to affect optimizer plans.

If you look at the stats accuracy along running ANALYZE TABLE after initial table population and when there are significant changes makes sense. For Innodb as index stats are computed first time table is accessed after restart this often means "never" because MySQL servers are restarted frequently enough. Even once per 3 months is often enough for many workloads. Add to this Innodb stats are less accurate by nature which means you can allow more data change while your
index stats remain as good as new.

Looking at stats accuracy is however a wrong way to look at the problem. Your index stats are a bit off, so what ? What really matters is not how accurate stats are but how good plans you're getting for your queries. If you're getting as good plans as with perfect stats why bother updating them ?
Also note many simple "queries" (using constants for index accesses) will not use index cardinality data at all but will estimate number of rows during query execution.

I typically look at ANALYZE TABLE and adding it to the table if I see having it run helps to get good plans. If query plans are good or bad independently of it being run there is need to bother - for bad plans use FORCE INDEX or change the query and report MySQL Optimizer Bug

But now lets see in the difference of behavior of ANALYZE TABLE for MyISAM vs Innodb.

I used the following simple table for tests:

I have populated it with data with following true cardinality:

Lets see how stats look for MYISAM:

Aha as you can see there is no cardinality stored with table as ANALYZE did not run yet.

As you can see after running ANALYZE we have exact cardinality for i and c columns, with cardinality for the pair (c,i) looks a bit off but is within 0.5% of the correct value so we can count on MyISAM values as almost exact.

As you see ANALYZE table tool a little bit of time to run (even for this very small table) this is because ANALYZE does index scans to find number of exact values in the table.

Now let us populate antest_innodb table which is same but uses Innodb format:

Very interesting result - after loading the data with INSERT in Innodb table we do not get NULL cardinality as with MyISAM but instead we get very wrong cardinality which shows us index prefix is unique (245900 is estimate for the row count in the table)

It is worth to note if you do ALTER TABLE Innodb, same as MyISAM will internally run analyze as soon as table is rebuilt and values will be more sensible:

Note however how much are these values off from reality. The "i" key cardinality is overestimated 3 times, "c" key prefix cardinality is underestimated 5 times and the combined (c,i) key cardinality is overestimated 2 times. So Innodb stats are are very inexact. Fortunately for most queries which use these stats accuracy at the order of magnitude is enough. Sometimes it is not and you're thinking why a hell it could be picking this strange plan.

Let us run ANALYZE TABLE for Innodb couple of more times to see how values change:

As we see subsequent runs change stats dramatically. For c prefix we got value changed to become 15 times larger. So Innodb stats are both inexact and unstable. So restarting server with Innodb may change stats dramatically and affect some query plans. You also may be getting different plans on different slaves with same data.

Another difference when it comes from handling the statistics comes from NULL handling.
MyISAM has a special variable which controls if NULLs should be considered equal when computing stats:

Too see the difference let me set column "c" to NULL in both tables and see how values change:

As you can see MyISAM set cardinality for prefix (c) and key(c,i) approximately to number of rows in the table treating all nulls different values. Innodb on the contrary treats all NULL values the same so
cardinality for (c) and (c,i) dropped significantly.

This means Innodb and MyISAM have different stats computation method by default.

Lets check how stats change for MyISAM if we change the stats computation method:

oops. Little gotcha. MySQL considers table up to date even though stats stored were computed with different method. If your table is written to actively you should not have this problem; I just did couple of updates to refresh update time.

So with nulls_equal method we see very different picture. It is considered we only have one distinct value for "c" and there are 101 distict values for (c,i) which is the same as value of distinct values in i column. These stats look much closer to what we get for Innodb table with same data though we can see Innodb stats are a bit off from reality too.

MySQL version note: This is from MySQL 5.0.62 if there are other versions which show different behavior.

Entry posted by peter | One comment

Add to: | | | |

Attention MySQL engineers and Drizzle contributors: upgrade to Bazaar 1.6.1 now to get some fairly massive performance speedups for bzr branch commands. As I noted in my last article on Launchpad code management, Bazaar 1.5 was having some performance issues when branching large project trees such as MySQL. In the article, I showed it was taking Bazaar 1.5 91 minutes to do the initial branch. With John Arbash Meinel's performance patches, the time to branch was cut down to 23 minutes, which is a fantastic improvement.

Ubuntu users: grab the 1.6.1 Bazaar package for your Ubuntu version from the Launchpad.net Bazaar Project Package Archive.

Mac OSX users:: grab the disk image here, with these instructions.

Windows users: grab the Windows installer for 1.6.1rc from the Launchpad repository.

We've launched this year’s AWS Start-Up Challenge, a contest for entrepreneurs and start-ups that will award the winner $50,000 in cash, $50,000 in AWS credits, a potential investment offer from Amazon.com, and more. Submissions will be accepted until October 3, 2008.

Learn more about the contest and enter your great idea today.

-- Jeff;