选择一个键值存储系统(Cassandra Vs Voldemort)
By Diego Erdody on May 07, 2010 Translated by Jametong

目的

在Medallia,我们的系统目前有一个关键组件是运行在一个开源的关系型数据库上.由于此组件主要通过主键来查询数据库的条目,我们想尝试将此组件切换到一个键值存储系统上,以利用键值系统提供的多种好处,包含分布式复制、负载均衡以及失败切换.对此组件进行重构以实现纵向扩展是我们的一个目标,附带的其它好处是,可以缓解我们目前较高的磁盘存储需求.

最近,我们花了部分时间来研究这项技术(以及部分其他技术改进,Medallia激动人心的时刻!),考察了多个不同选项.长话短说,最终落在以下两个选择上:Apache Cassandra与Project Voldemort.

这两个项目看似是他们所在开源类别中最成熟的了,都可以提供内置的分散化集群支持,包含分区、容错性以及高可用性.两者都是基于Amazon的Dynamo论文,主要的差异是,Voldemort遵循简单的键值模型,而Cassandra使用了基于BigTable持久化模型的面向列的模型.两者都支持读一致性,也就是读操作总是返回最新的数据,这一点是我们业务所需要的.

高层次的比较

Project Voldemort

虽然不是一份详尽的清单,下面是我们考查这两个存储系统最关心的优势与劣势.

• 优势
• 更简单的API
• 基于Berkley DB的持久化,一个成熟并广泛使用的键值DB
• 使用向量时钟而不是简单的时间戳.它不需要节点(客户端)的时钟保持同步.
• 劣势
• 没有内置的”多数据中心”相关路由支持(意味着至少有一个额外的数据中心有此数据的一份拷贝)

Apache Cassandra

• 优势
• 更广泛的生产系统部署(Facebook、Twitter、Digg、Rackspace)
• 更丰富的API,值可以支持动态的列结构(Schema-free).列可以独立演化,意味着你不需要读出整个结构就可以更新其中的一列.
• 为写操作做过专门优化(设计上).
• 可配置的一致性级别(在每个请求上指定)
• 劣势
• 文件格式仍在开发中,内部结构仍然可能会发生变化.鉴于它所支持的灵活性,文件格式更加复杂也更加难以理解,在性能方面尤其如此
• 需要同步时钟(NTP)(节点与客户端都需要)
• 与竞争产品相比,读操作更加磁盘密集
• 不支持客户端的冲突检测,因此最近的数据总是赢家

性能测试

令我们吃惊的是,这是我们找到的对这两个项目的进行性能比较的唯一链接,因此,我们决定写这篇文章来分享我们的研究.我们使用了vpork的测试框架,对它的代码做了修改以适应我们的需求,升级客户端代码到最新版本、添加热身阶段、增加了重写能力。下面是我们测试的结果.

配置:

• 版本
• Voldemort v0.80.1
• Cassandra 0.6.0-beta3
• 机器-3个类似与如下配置的节点
• 最大4GB的堆大小(heap size)
• 复制参数: N=3(每个条目的副本数),R=2(每次读时需要等待返回的节点数),W=2(每次写需要等待响应的节点数)
• 每台服务器上有8个处理器(Intel(R) Xeon(R) CPU E5504 @2.00GHz)
• 1TB的磁盘空间(Seagate ST31000340NS,7200rpm,32MB Cache)
• 持久化参数
• Voldemort(默认值)
• Key-serializer: String
• Value-serializer : identity(字节数组)
• persistence=bdb(Berkley DB)
• Cassandra
• ColumnFamily定义: CompareWith=”BytesType” RowsCached=”10000″
• ReplicationFactor=3
• Partitioner=org.apache.cassandra.dht.RandomPartitioner
• ConcurrentReads=16
• ConcurrenWrites=32
• 测试
• 客户端线程数: 40
• 初始加载:500万记录-每次测试开始前就有的记录数
• 热身:2万记录-在记录测试时间前的初始化写操作
• 每次测试的操作次数:50万

我们测试以下4种不同的写-重写-读配置.一次写操作等价于一个包含一条新记录(不存在的Key)的put操作.一次重写操作是一个包含已有Key的put操作.一次读是对一个已有Key的get操作.下面是我们测试的配置:

• 50% Write 50% Read
• 10% Write 40% Rewrite 50% Read
• 50% Rewrite 50% Read
• 90% Rewrite 10% Read

所有这些测试,我们都测试两组不同的Value大小,15KB与1.5KB.虽然我们评估了不同的选项,但是,对于我们的需求来讲,最后一种配置以及15KB的数据条目大小才是最具代表性的场景.

第一组图表展示延时(Latency),或者说是每个读写操作在每种情况下完成所需的平均时间.值越低越好.与预期一致,Cassandra的写操作(或重写)时间总是优于Voldemort,随着场景的不同读操作时间有点变化,不过总体上两者相差不大.

第二组图显示99%条件下的最大延时时间;仍然是值越小越好:

在前端,我们有一个写回高速缓存,这意味着写操作不会影响用户体验.另一方面,读操作直接与页面加载有关系.这也是为什么我们特别关注Cassandra在最后一个场景中处理15KB数据的峰值时间.我们还做了一些更进一步的测试,来度量99.9%与99.99%条件下的情况,差异更加明显:在99.9%时,Cassandra需要5050ms,Voldemort需要748ms,在99.99%时,Cassandra需要9176ms,Voldemort需要1129ms.这个巨大的差异是影响我们决策的一个关键指标.

最后这两个图表展示每秒操作数(读/写)的吞吐量.在这两个图中,值越大越好:

附注:

• Cassandra建议将Commit log与数据目录分别存储在不同的磁盘上以提高性能,我们的测试使用的都是同一块磁盘.

在测试过程中发现的问题:

• 如果在不同的线程中调用同一个Key,Voldemort的client.put(K key,V value)(不是取出特定版本对象的那个方法)会抛出异常ObsoleteVersionException. javadoc的表述是”关联这个Key的给定值,覆盖这个key之前已经存储的任何值”,因此这个错误是预料之外的.

最后的赢家是…

大体上,我认为没有明确的赢家.最佳选择依赖于多个因素,这需要每个公司自己去评估.我的偏好也在考查与测试过程中发生了多次变化.

已经说过,我们必须做出选择,并且最后决定选择使用Project Voldemort.主要的原因是简单、更好的版本控制、成熟的持久化层,以及延时
的可预测性.

我们现在已经开始开发新的解决方案,在我们将其应用到生产环境可能还需要一段时间,但是,我们想要将我们的初步成果与所有考虑这两个选项之一的人共享,这样,他们在做决策的时候就可以多一个参考.

我将对持续关注它的走向.

Diego Erdody
Lead Software Engineer

其他比较有用的关于比较不同的键值存储的文章:

• http://blog.endpoint.com/2010/03/nosql-live-dynamo-derivatives-cassandra.html
• http://www.vineetgupta.com/2010/01/nosql-databases-part-1-landscape.html
• http://arstechnica.com/business/data-centers/2010/02/-since-the-rise-of.ars/2
• http://www.rackspacecloud.com/blog/2009/11/09/nosql-ecosystem/
• http://bhavin.directi.com/tag/memcached/
• http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/

Related posts:

1. Cassandra – 一个分散的非结构化存储系统

I recently reformatted my laptop with the latest Ubuntu LTS release, 10.04, aka Lucid Lynx. Since I like to have a native client installation as well as a portable sandbox server, I decided to install the latest version of Oracle EE, 11.2.0.1.

Rather than re-invent the wheel, I’m going to direct you to the previous Oracle-on-Ubuntu post by my colleague Augusto Bott. Many of the directions there hold true here (even with 32-bit vs 64-bit), with a few exceptions.

Download the Software
First and foremost, download the 32-or-64-bit installer files (there are 2 of them) from OTN.

Installing Pre-requisite Packages
Install these software packages on your Ubuntu 10.04 system:

$ sudo apt-get install unzip build-essential x11-utils rpm ksh lsb-rpm libaio1

Ubuntu 10.04 comes with libstdc++6 installed. However, Oracle 11gR2 requires libstdc++5. If you do not install libstdc++5, you will see errors as described in this OTN thread. The fix, as described in that thread, is to download and manually shoehorn the libstdc++5 library files onto Ubuntu 10.04:

$ wget http://mirrors.kernel.org/ubuntu/pool/universe/g/gcc-3.3/libstdc++5_3.3.6-17ubuntu1_amd64.deb
$ dpkg-deb -x libstdc++5_3.3.6-17ubuntu1_amd64.deb ia64-libs
$ sudo cp ia64-libs/usr/lib/libstdc++.so.5.0.7 /usr/lib64/
$ cd /usr/lib64/
$ sudo ln -s libstdc++.so.5.0.7 libstdc++.so.5

$ wget http://security.ubuntu.com/ubuntu/pool/universe/i/ia32-libs/ia32-libs_2.7ubuntu6.1_amd64.deb
$ dpkg-deb -x ia32-libs_2.7ubuntu6.1_amd64.deb ia32-libs
$ sudo cp ia32-libs/usr/lib32/libstdc++.so.5.0.7 /usr/lib32/
$ cd /usr/lib32
$ sudo ln -s libstdc++.so.5.0.7 libstdc++.so.5

Now we’re ready to move on.

Create User and Groups
First create the oinstall and dba groups:

$ sudo su -
# addgroup oinstall
# addgroup dba

Then create the oracle user and assign it to those groups:

# useradd -g oinstall -G dba -p password -d /home/oracle -s /bin/bash oracle
# mkdir /home/oracle
# chown -R oracle:dba /home/oracle

Now we create some symbolic links to give the Ubuntu system a more “Red Hat-ish layout,” as Augusto put it:

# ln -s /usr/bin/awk /bin/awk
# ln -s /usr/bin/rpm /bin/rpm
# ln -s /usr/bin/basename /bin/basename
# mkdir /etc/rc.d
# for i in 0 1 2 3 4 5 6 S ; do ln -s /etc/rc$i.d /etc/rc.d/rc$i.d ; done
# mkdir -p /u01/app/oracle
# chown -R oracle:dba /u01

Then we update some sysctl parameters by editing /etc/sysctl.conf. It is wisest to backup this file first and then add these lines to the end of /etc/sysctl.conf:

fs.file-max = 6815744
fs.aio-max-nr = 1048576
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 1048576
net.core.wmem_max = 1048576
net.ipv4.ip_local_port_range = 9000 65535

Note that these values are different from what was needed in 11gR1.

Now we update some limits for the oracle software owner by adding these lines to the end of /etc/security/limits.conf:

oracle soft nproc 2047
oracle hard nproc 16383
oracle soft nofile 1023
oracle hard nofile 65535

Again, it is best to backup the file first.

Now we activate the new settings from /etc/sysctl.conf with this command:

# sysctl -p

You should see the new settings in the output of that command.

Unpack and Install the Software
From here we go into the directory containing our two 11gR2 database zip files. I’m using the 64-bit edition, so my files are named linux.x64_11gR2_databaseXof2.zip, where X is 1 or 2. I unzip both of these files, which creates a “database” subdirectory:

$ unzip linux.x64_11gR2_database_1of2.zip
$ unzip linux.x64_11gR2_database_2of2.zip

Each of these commands will produce a large volume of output to the terminal as it lists each file being unpacked. Once this is done, your directory should look something like this:

$ ls -lh
total 2.2G
drwxr-xr-x 8 seiler seiler 4.0K 2009-08-20 14:34 database
-rw-r--r-- 1 seiler seiler 1.2G 2010-03-09 11:33 linux.x64_11gR2_database_1of2.zip
-rw-r--r-- 1 seiler seiler 1.1G 2010-03-09 12:38 linux.x64_11gR2_database_2of2.zip

Now you simply cd into the database directory and run the runInstaller program:

$ cd database
$ ./runInstaller

This will launch the Oracle Universal Installer, or OUI, program. The rest is pretty straight forward. Since this is just a sandbox, I chose not to provide any email info and declined to receive updates. I chose to first install the software only, which went perfectly well. OUI will complain about missing packages, since it is checking for RPMs. We can safely ignore these and proceed with installation.

Once installation is done I set these variables in my bash environment:

export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1
export ORACLE_SID=orcl
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$PATH
export EDITOR=/usr/bin/vi

Then some quick tests to verify installation:

$ sqlplus /nolog

SQL*Plus: Release 11.2.0.1.0 Production on Thu Jun 10 23:46:42 2010

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

SQL> quit
$ /u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch lsinventory
Invoking OPatch 11.1.0.6.6

Oracle Interim Patch Installer version 11.1.0.6.6
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.6
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.2.0/dbhome_1/oui
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2010-06-10_23-46-53PM.log

Patch history file: /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2010-06-10_23-46-53PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1): 

Oracle Database 11g                                                  11.2.0.1.0
There are 1 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

--------------------------------------------------------------------------------

OPatch succeeded.

Everything looks hunky-dory. Let’s create an instance.

I launched dbca to create an Oracle 11gR2 instance. The one problem I encountered here was that, at the end of dbca prompts, the “Confirmation” dialog window was blank. Other blog posts on the internet suggested blindly clicking around until you magically hit the “OK” button, but more often than not I would hit the “Cancel” button. Turns out this is a not-so-uncommon problem with Java Swing and Gnome’s visual effects. The solution for Gnome users is to disable Visual Effects in the Appearance preferences before launching dbca. If you are using a lightweight desktop environment such as LXDE, you shouldn’t have this problem.

And that’s that. The rest is nothing new to those who have done installations before. You can optionally configure TNS names or Listener with the netca tool, or log into your new instance and enjoy.

Special thanks to Augusto Bott for not only authoring the previous Oracle-on-Ubuntu articles but also for suggesting the Gnome Visual Effects conflict!

这个问题是在作Online redef测试的时候碰见的，如果在primary执行START_REDEF_TABLE的时候，写错了目标表的名字，可能导致在logical standby上报这个错误然后导致apply停掉。并且该错误无法被skip。貌似bug.

SQL> create table large(id number primary key,text varchar(32) ) tablespace users;
Table created.

SQL> insert into large select object_id,object_name from dba_objects where object_id is not null;
72631 rows created.

SQL> create table TEST_NEW(id number primary key,text varchar(32) ) tablespace users2;
Table created.

SQL> exec dbms_redefinition.can_redef_table ( USER,’TEST’, DBMS_REDEFINITION.CONS_USE_PK) ;
PL/SQL procedure successfully completed.

然后执行START_REDEF_TABLE;如下敲错成“-”，自然会在primary上提示表large-new不存在。

exec dbms_redefinition.START_REDEF_TABLE ( USER, ‘LARGE’, ‘LARGE-NEW’, NULL, DBMS_REDEFINITION.CONS_USE_PK);

然后会在logical standby上碰见ORA-26808错误。

ORA-26808: Apply process AS00 died unexpectedly.
ORA-23481: unexpected name string “LARGE-NEW”

下面是logical standby上的日志，

Then on logical standby, when it parse “LARGE-NEW”, apply process died and report ORA-23481: unexpected name string “LARGE-NEW”.

————–alter.log on logical standby ———————

ALTER DATABASE START LOGICAL STANDBY APPLY IMMEDIATE
ALTER DATABASE START LOGICAL STANDBY APPLY (oradb)
with optional part
IMMEDIATE
Attempt to start background Logical Standby process
LOGSTDBY parameters set by user:
LOGSTDBY MAX_EVENTS_RECORDED = 2000000000
LOGSTDBY RECORD_UNSUPPORTED_OPERATIONS = TRUE
LOGSTDBY LOG_AUTO_DELETE = FALSE
LOGSTDBY PRESERVE_COMMIT_ORDER = FALSE
Thu Jun 03 08:26:11 2010
LSP0 started with pid=30, OS id=9363
Completed: ALTER DATABASE START LOGICAL STANDBY APPLY IMMEDIATE
LOGMINER: Parameters summary for session# = 5
LOGMINER: Number of processes = 3, Transaction Chunk Size = 201
LOGMINER: Memory Size = 30M, Checkpoint interval = 150M
LOGMINER: SpillScn 7930843, ResetLogScn 754488
LOGMINER: summary for session# = 5
LOGMINER: StartScn: 0 (0×0000.00000000)
LOGMINER: EndScn: 0 (0×0000.00000000)
LOGMINER: HighConsumedScn: 7930824 (0×0000.007903c8)
LOGMINER: session_flag 0×1
Thu Jun 03 08:26:12 2010
LOGMINER: session#=5, builder MS01 pid=32 OS id=9367 sid=34 started
Thu Jun 03 08:26:13 2010
LOGMINER: session#=5, reader MS00 pid=31 OS id=9365 sid=38 started
LOGMINER: Begin mining logfile for session 5 thread 1 sequence 321, /home/oracle/app/oradb/oradata/oradb/arch2/1_321_696990990.dbf
Thu Jun 03 08:26:13 2010
LSP2 started with pid=36, OS id=9371
Thu Jun 03 08:26:14 2010
LOGMINER: session#=5, preparer MS02 pid=35 OS id=9369 sid=51 started
Thu Jun 03 08:26:15 2010
LOGSTDBY Analyzer process AS00 started with server id=0 pid=37 OS id=9373
Thu Jun 03 08:26:15 2010
LOGSTDBY Apply process AS01 started with server id=1 pid=38 OS id=9375
Thu Jun 03 08:26:15 2010
LOGSTDBY Apply process AS03 started with server id=3 pid=36 OS id=9379
Thu Jun 03 08:26:15 2010
LOGSTDBY Apply process AS02 started with server id=2 pid=39 OS id=9377
Thu Jun 03 08:26:15 2010
LOGSTDBY Apply process AS04 started with server id=4 pid=42 OS id=9381
Thu Jun 03 08:26:16 2010
LOGSTDBY Apply process AS05 started with server id=5 pid=44 OS id=9383
LOGSTDBY Analyzer process AS00 server id=0 pid=37 OS id=9373 stopped
Errors in file /home/oracle/app/oradb/diag/rdbms/standby/oradb/trace/oradb_lsp0_9363.trc:
ORA-26808: Apply process AS00 died unexpectedly.
ORA-23481: unexpected name string “LARGE-NEW”
LOGSTDBY Apply process AS01 server id=1 pid=38 OS id=9375 stopped
LOGSTDBY Apply process AS02 server id=2 pid=39 OS id=9377 stopped
LOGSTDBY Apply process AS03 server id=3 pid=36 OS id=9379 stopped
LOGSTDBY Apply process AS04 server id=4 pid=42 OS id=9381 stopped
LOGSTDBY Apply process AS05 server id=5 pid=44 OS id=9383 stopped
Errors in file /home/oracle/app/oradb/diag/rdbms/standby/oradb/trace/oradb_as00_9373.trc:
ORA-23481: unexpected name string “LARGE-NEW”
LOGMINER: session#=5, reader MS00 pid=31 OS id=9365 sid=38 stopped
LOGMINER: session#=5, builder MS01 pid=32 OS id=9367 sid=34 stopped
LOGMINER: session#=5, preparer MS02 pid=35 OS id=9369 sid=51 stopped
LOGSTDBY status: ORA-16222: automatic Logical Standby retry of last action
LOGMINER: Parameters summary for session# = 5
LOGMINER: Number of processes = 3, Transaction Chunk Size = 201
LOGMINER: Memory Size = 30M, Checkpoint interval = 150M
LOGMINER: SpillScn 7930843, ResetLogScn 754488
LOGMINER: summary for session# = 5
LOGMINER: StartScn: 0 (0×0000.00000000)
LOGMINER: EndScn: 0 (0×0000.00000000)
LOGMINER: HighConsumedScn: 7930824 (0×0000.007903c8)
LOGMINER: session_flag 0×1
Thu Jun 03 08:26:18 2010
LOGMINER: session#=5, reader MS00 pid=31 OS id=9385 sid=38 started
Thu Jun 03 08:26:19 2010
LOGMINER: session#=5, builder MS01 pid=32 OS id=9387 sid=51 started
Thu Jun 03 08:26:19 2010
LOGMINER: session#=5, preparer MS02 pid=35 OS id=9389 sid=43 started
LOGMINER: Begin mining logfile for session 5 thread 1 sequence 321, /home/oracle/app/oradb/oradata/oradb/arch2/1_321_696990990.dbf
Thu Jun 03 08:26:20 2010
LOGSTDBY Apply process AS01 started with server id=1 pid=37 OS id=9393
Thu Jun 03 08:26:20 2010
LOGSTDBY Apply process AS02 started with server id=2 pid=38 OS id=9395
Thu Jun 03 08:26:20 2010
LOGSTDBY Apply process AS03 started with server id=3 pid=39 OS id=9397
Thu Jun 03 08:26:20 2010
LOGSTDBY Analyzer process AS00 started with server id=0 pid=36 OS id=9391
Thu Jun 03 08:26:20 2010
LOGSTDBY Apply process AS04 started with server id=4 pid=42 OS id=9399
Thu Jun 03 08:26:20 2010
LOGSTDBY Apply process AS05 started with server id=5 pid=44 OS id=9401
LOGSTDBY Analyzer process AS00 server id=0 pid=36 OS id=9391 stopped
Thu Jun 03 08:26:21 2010
Errors in file /home/oracle/app/oradb/diag/rdbms/standby/oradb/trace/oradb_lsp0_9363.trc:
ORA-26808: Apply process AS00 died unexpectedly.
ORA-23481: unexpected name string “LARGE-NEW”
LOGSTDBY Apply process AS01 server id=1 pid=37 OS id=9393 stopped
LOGSTDBY Apply process AS02 server id=2 pid=38 OS id=9395 stopped
LOGSTDBY Apply process AS03 server id=3 pid=39 OS id=9397 stopped
LOGSTDBY Apply process AS04 server id=4 pid=42 OS id=9399 stopped
LOGSTDBY Apply process AS05 server id=5 pid=44 OS id=9401 stopped
Errors in file /home/oracle/app/oradb/diag/rdbms/standby/oradb/trace/oradb_as00_9391.trc:
ORA-23481: unexpected name string “LARGE-NEW”
LOGMINER: session#=5, reader MS00 pid=31 OS id=9385 sid=38 stopped
LOGMINER: session#=5, builder MS01 pid=32 OS id=9387 sid=51 stopped
LOGMINER: session#=5, preparer MS02 pid=35 OS id=9389 sid=43 stopped
———————————————————————————————–

cat /home/oracle/app/oradb/diag/rdbms/standby/oradb/trace/oradb_as00_9391.trc

*** 2010-06-03 08:26:22.065
ORA-23481: unexpected name string “LARGE-NEW”
OPIRIP: Uncaught error 447. Error stack:
ORA-00447: fatal error in background process
ORA-23481: unexpected name string “LARGE-NEW”

—————————————————————————————————

cat /home/oracle/app/oradb/diag/rdbms/standby/oradb/trace/oradb_lsp0_9363.trc

*** 2010-06-03 08:26:21.980
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0×0, level=0, mask=0×0)
—– Error Stack Dump —–
ORA-26808: Apply process AS00 died unexpectedly.
ORA-23481: unexpected name string “LARGE-NEW”
KNACDMP: *******************************************************
KNACDMP: Dumping apply coordinator’s context at bfc99c94
KNACDMP: Apply Engine # 0
KNACDMP: Apply Engine name
KNACDMP: Coordinator’s Watermarks ——————————
KNACDMP: Apply High Watermark = 0000007903c8 (SCN=0×0000.007903c8)
KNACDMP: Apply Low Watermark = 0000007903c8 (SCN=0×0000.007903c8)
KNACDMP: Recovery Low Watermark = 0000007903c8 (SCN=0×0000.007903c8)
KNACDMP: Fetch Low Watermark = (SCN=0×0000.00000000)
KNACDMP: Fetch Low Watermark Time = 0
KNACDMP: Oldest SCN = (SCN=0×0000.00000000)
KNACDMP: Oldest XID =
KNACDMP: Oldest Create Time = 0
KNACDMP: Last replicant syncpoint SCN = 0×0000.00000000
KNACDMP: Last syncpoint at primary SCN = 0×0000.00000000
KNACDMP: First partition max pos = 000000799b61 (SCN=0×0000.00799b61)
KNACDMP: Last partition max pos = 000000799b61 (SCN=0×0000.00799b61)
KNACDMP: Last processed = 0000007903d7 (SCN=0×0000.007903d7)
KNACDMP: Conservative pos = 0000007903d7 (SCN=0×0000.007903d7)
KNACDMP: Recovery start pos = (SCN=0×0000.00000000)
KNACDMP: Recovery high watermark = 0000007903c8 (SCN=0×0000.007903c8)
KNACDMP: Spill LWM = (SCN=0×0000.00000000)
KNACDMP: Spill LWM Create Time = 0
KNACDMP: Coordinator’s constants ——————————-
KNACDMP: number of apply slaves = 5
KNACDMP: safety level (K) = 1
KNACDMP: max txns in memory = 1200
KNACDMP: max constraints per table = 653
KNACDMP: hash table size (in entries) = 1000000
KNACDMP: Coordinator’s intervals ——————————-
KNACDMP: syncpoint interval (ms) = 4294967295
KNACDMP: write low watermark interval(ms)= 1
KNACDMP: Coordinator’s timers/counters ————————-
KNACDMP: current time = 1275524781
KNACDMP: low watermark timer = 0
KNACDMP: syncpoint timer = 1275524780
KNACDMP: txnbufsize timer = 1275524780
KNACDMP: Coordinator’s txn counts ————————-
KNACDMP: total txns applied = 0
KNACDMP: number of unassigned txns = 0
KNACDMP: avg number of unassigned txns = 0.00
KNACDMP: total applied at last plwm write= 0
KNACDMP: apply prog. entries below plwm = 0
KNACDMP: total unassigned lcrs = 0
KNACDMP: Coordinator’s State/Flags —————————–
KNACDMP: Coordinator’s State = KNACST_APPLY_UNTIL_END
KNACDMP: Coordinator’s Flags = 0×8200
KNACDMP: Slave counts ——————————————
KNACDMP: number of reserved slaves = 1
KNACDMP: number of admin slaves = 0
KNACDMP: number of slaves in wait cmt = 0
KNACDMP: number of slaves suspended = 0
KNACDMP: number of safe slaves = 0
KNACDMP: avg number of idle slaves = 0.00
KNACDMP: number of slaves initializing = 0
KNACDMP: number of slaves terminating = 0
KNACDMP: Slave Lists ——————————————-
KNACDMP: Dumping All Slaves :-
Slave id = 0, State = 8, Flags = 0, Not Assigned
Slave id = 1, State = 0, Flags = 0, Not Assigned
Slave id = 2, State = 0, Flags = 0, Not Assigned
Slave id = 3, State = 0, Flags = 0, Not Assigned
Slave id = 4, State = 0, Flags = 0, Not Assigned
Slave id = 5, State = 0, Flags = 0, Not Assigned
KNACDMP: End dumping all slaves
KNACDMP: syncdep slaves = { }
KNACDMP: cont chunk slaves = { }
KNACDMP: cont slaves = { }
KNACDMP: exec txn slaves = { }
KNACDMP: Active slaves (0) = { }
KNACDMP: Idle slaves (5) = { 1 2 3 4 5 }
KNACDMP: Txn Lists ———————————————
KNACDMP: Dumping all txns :-
KNACDMP: End dumping all txns.
KNACDMP: Complete txns = { ** NO UNASS ** }
KNACDMP: Unassigned txns = { }
KNACDMP: *******************************************************

*** 2010-06-03 08:26:25.156
Warning: Apply error received: ORA-26714: User error encountered while applying. Clearing.
=====================

事件视图DBA_LOGSTDBY_EVENTS中没有记录任何有用的信息。
如下等各种方法都无法skip该错误, SQL apply进程无法启动。

exec dbms_logstdby.SKIP_ERROR(’TABLE’,'BINZHANG’,'%’,null);
exec dbms_logstdby.SKIP_ERROR(’NON_SCHEMA_DDL’,'BINZHANG’,'LARGE_NEW’,null);
exec dbms_logstdby.SKIP_ERROR(’SCHEMA_DLL’,'BINZHANG’,'LARGE_NEW’,null);
EXECUTE DBMS_LOGSTDBY.SKIP (stmt => ‘DML’, schema_name => ‘BINZHANG’, object_name => ‘%’, proc_name => null);
EXECUTE DBMS_LOGSTDBY.SKIP (stmt => ‘SCHEMA_DLL’, schema_name => ‘BINZHANG’, object_name => ‘%’, proc_name => null);
EXECUTE DBMS_LOGSTDBY.SKIP (stmt => ‘NON_SCHEMA_DDL’, schema_name => ‘BINZHANG’, object_name => ‘%’, proc_name => null);

测试环境:oracle11gR2 redhat5.

Understanding the basic concepts behind MapReduce is not a very difficult task, but those using extensively MapReduce tasks inside Hadoop are already facing new challenges like:

• how can you run multiple map and/or reduce phases in your data processing?
• how can you better coordinate the data processing execution flow for more complex scenarios?
• how can you perform additional work between map/reduce phases?

Addressing these new challenges is the goal of the ☞ Cascading project:

Cascading is a feature rich API for defining and executing complex, scale-free, and fault tolerant data processing workflows on a Hadoop cluster.

Christopher Curtin’s slides embedded below are offering a good overview of what can be achieved using Cascading (starting with slide 20).

Just one day after mentioning Giazzard as a possible solution for partitioning data in CouchDB, I’m hearing that Twitter is experimenting with Redis and uses ☞ Gizzard for data partitioning. Code for this experiment is available on ☞ GitHub under the name Haplocheirus^[1]