diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..8dbcd50 --- /dev/null +++ b/.gitignore @@ -0,0 +1,38 @@ +target/ +.idea/ +*.iml +out/ +gen/ +idea-gitignore.jar +resources/templates.list +resources/gitignore/* +build/ +build.properties +junit*.properties +IgnoreLexer.java~ +.gradle +/verification + +# Compiled class file +*.class + +# Log file +*.log + +# BlueJ files +*.ctxt + +# Mobile Tools for Java (J2ME) +.mtj.tmp/ + +# Package Files # +*.jar +*.war +*.nar +*.ear +*.zip +*.tar.gz +*.rar + +# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml +hs_err_pid* diff --git a/LICENSE.txt b/LICENSE.txt new file mode 100644 index 0000000..440b4d3 --- /dev/null +++ b/LICENSE.txt @@ -0,0 +1,183 @@ +Apache License +Version 2.0, January 2004 +http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + +"License" shall mean the terms and conditions for use, reproduction, and +distribution as defined by Sections 1 through 9 of this document. + +"Licensor" shall mean the copyright owner or entity authorized by the +copyright owner that is granting the License. + +"Legal Entity" shall mean the union of the acting entity and all other +entities that control, are controlled by, or are under common control with +that entity. For the purposes of this definition, "control" means (i) the +power, direct or indirect, to cause the direction or management of such +entity, whether by contract or otherwise, or (ii) ownership of fifty percent +(50%) or more of the outstanding shares, or (iii) beneficial ownership of such +entity. + +"You" (or "Your") shall mean an individual or Legal Entity exercising +permissions granted by this License. + +"Source" form shall mean the preferred form for making modifications, +including but not limited to software source code, documentation source, and +configuration files. + +"Object" form shall mean any form resulting from mechanical transformation or +translation of a Source form, including but not limited to compiled object +code, generated documentation, and conversions to other media types. + +"Work" shall mean the work of authorship, whether in Source or Object form, +made available under the License, as indicated by a copyright notice that is +included in or attached to the work (an example is provided in the Appendix +below). + +"Derivative Works" shall mean any work, whether in Source or Object form, that +is based on (or derived from) the Work and for which the editorial revisions, +annotations, elaborations, or other modifications represent, as a whole, an +original work of authorship. For the purposes of this License, Derivative +Works shall not include works that remain separable from, or merely link (or +bind by name) to the interfaces of, the Work and Derivative Works thereof. + +"Contribution" shall mean any work of authorship, including the original +version of the Work and any modifications or additions to that Work or +Derivative Works thereof, that is intentionally submitted to Licensor for +inclusion in the Work by the copyright owner or by an individual or Legal +Entity authorized to submit on behalf of the copyright owner. For the purposes +of this definition, "submitted" means any form of electronic, verbal, or +written communication sent to the Licensor or its representatives, including +but not limited to communication on electronic mailing lists, source code +control systems, and issue tracking systems that are managed by, or on behalf +of, the Licensor for the purpose of discussing and improving the Work, but +excluding communication that is conspicuously marked or otherwise designated +in writing by the copyright owner as "Not a Contribution." + +"Contributor" shall mean Licensor and any individual or Legal Entity on behalf +of whom a Contribution has been received by Licensor and subsequently +incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of this +License, each Contributor hereby grants to You a perpetual, worldwide, +non-exclusive, no-charge, royalty-free, irrevocable copyright license to +reproduce, prepare Derivative Works of, publicly display, publicly perform, +sublicense, and distribute the Work and such Derivative Works in Source or +Object form. + +3. Grant of Patent License. Subject to the terms and conditions of this +License, each Contributor hereby grants to You a perpetual, worldwide, +non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this +section) patent license to make, have made, use, offer to sell, sell, import, +and otherwise transfer the Work, where such license applies only to those +patent claims licensable by such Contributor that are necessarily infringed by +their Contribution(s) alone or by combination of their Contribution(s) with +the Work to which such Contribution(s) was submitted. If You institute patent +litigation against any entity (including a cross-claim or counterclaim in a +lawsuit) alleging that the Work or a Contribution incorporated within the Work +constitutes direct or contributory patent infringement, then any patent +licenses granted to You under this License for that Work shall terminate as of +the date such litigation is filed. + +4. Redistribution. You may reproduce and distribute copies of the Work or +Derivative Works thereof in any medium, with or without modifications, and in +Source or Object form, provided that You meet the following conditions: + +(a) You must give any other recipients of the Work or Derivative Works a copy +of this License; and + +(b) You must cause any modified files to carry prominent notices stating that +You changed the files; and + +(c) You must retain, in the Source form of any Derivative Works that You +distribute, all copyright, patent, trademark, and attribution notices from the +Source form of the Work, excluding those notices that do not pertain to any +part of the Derivative Works; and + +(d) If the Work includes a "NOTICE" text file as part of its distribution, +then any Derivative Works that You distribute must include a readable copy of +the attribution notices contained within such NOTICE file, excluding those +notices that do not pertain to any part of the Derivative Works, in at least +one of the following places: within a NOTICE text file distributed as part of +the Derivative Works; within the Source form or documentation, if provided +along with the Derivative Works; or, within a display generated by the +Derivative Works, if and wherever such third-party notices normally appear. +The contents of the NOTICE file are for informational purposes only and do not +modify the License. You may add Your own attribution notices within Derivative +Works that You distribute, alongside or as an addendum to the NOTICE text from +the Work, provided that such additional attribution notices cannot be +construed as modifying the License. + +You may add Your own copyright statement to Your modifications and may provide +additional or different license terms and conditions for use, reproduction, or +distribution of Your modifications, or for any such Derivative Works as a +whole, provided Your use, reproduction, and distribution of the Work otherwise +complies with the conditions stated in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, any +Contribution intentionally submitted for inclusion in the Work by You to the +Licensor shall be under the terms and conditions of this License, without any +additional terms or conditions. Notwithstanding the above, nothing herein +shall supersede or modify the terms of any separate license agreement you may +have executed with Licensor regarding such Contributions. + +6. Trademarks. This License does not grant permission to use the trade names, +trademarks, service marks, or product names of the Licensor, except as +required for reasonable and customary use in describing the origin of the Work +and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or agreed to in +writing, Licensor provides the Work (and each Contributor provides its +Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied, including, without limitation, any warranties +or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A +PARTICULAR PURPOSE. You are solely responsible for determining the +appropriateness of using or redistributing the Work and assume any risks +associated with Your exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, whether in +tort (including negligence), contract, or otherwise, unless required by +applicable law (such as deliberate and grossly negligent acts) or agreed to in +writing, shall any Contributor be liable to You for damages, including any +direct, indirect, special, incidental, or consequential damages of any +character arising as a result of this License or out of the use or inability +to use the Work (including but not limited to damages for loss of goodwill, +work stoppage, computer failure or malfunction, or any and all other +commercial damages or losses), even if such Contributor has been advised of +the possibility of such damages. + +9. Accepting Warranty or Additional Liability. While redistributing the Work +or Derivative Works thereof, You may choose to offer, and charge a fee for, +acceptance of support, warranty, indemnity, or other liability obligations +and/or rights consistent with this License. However, in accepting such +obligations, You may act only on Your own behalf and on Your sole +responsibility, not on behalf of any other Contributor, and only if You agree +to indemnify, defend, and hold each Contributor harmless for any liability +incurred by, or claims asserted against, such Contributor by reason of your +accepting aense to your work, attach the following boilerplate notice, with +the fields enclosed by brackets "[]" replaced with your own identifying +information. (Don't include the brackets!) The text should be enclosed in the +appropriate comment syntax for the file format. We also recommend that a file +or class name and description of purpose be included on the same "printed +page" as the copyright notice for easier identification within third-party +archives. + +Copyright [yyyy] [name of copyright owner] + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + +See the License for the specific language governing permissions and +limitations under the License. + + + diff --git a/NOTICE.txt b/NOTICE.txt new file mode 100644 index 0000000..26b7c84 --- /dev/null +++ b/NOTICE.txt @@ -0,0 +1,10 @@ +======================================================================= +NOTICE file for use with, and corresponding to Section 4 of, +the Apache License, Version 2.0, +in this case for the Anarchy Ape project. +========================================================================= + + This product includes software developed by + Yahoo! Inc. (www.yahoo.com) + Copyright (c) 2010 Yahoo! Inc. All rights reserved. + diff --git a/README.md b/README.md new file mode 100644 index 0000000..a31cf9a --- /dev/null +++ b/README.md @@ -0,0 +1,172 @@ +AnarchyApe +========== + +Fault injection tool for hadoop cluster from yahoo anarchyape + + +Compilation +----------- +[Java] +Required version: 1.7.0.21 or later +``` +cd src/main/java +download log4j.ar and commons-cli.jar + +java -cp . ape/Main.java + +rm ape.jar + +jar cfm ape.jar META-INF/MANIFEST.MF ape META-INF/services org + +(old way) +javac -cp .:log4j-1.4.12.jar:commons-cli-1.2.jar ape/*.java +``` + +[Perl] +``` + perl Makefile.PL + cpan -i JSON + make + make test + make install +``` + +Running +------- +[Perl] +./ape.pl [remote_ip_list_file] + +[Java] +``` +java -jar ape.jar [commands] + +log file: /var/log/ape.log + +(old way) +java -cp .:log4j-1.4.12.jar ape/Main +``` + +(Local run) +``` +java -jar ape.jar -L -S 100 5 +``` +injects slow network with delay 100 milliseconds for 5 seconds. + +(Remote run) +Install pdsh: +``` +yum install pdsh +apt-get install pdsh + +java -jar ape.jar -R node1,node2,node3 -S 100 5 +creates a script to run on the remote hosts: +pdsh -Rssh -w node1,node2,node3 '/usr/local/bin/ape -L -S 100 5' +``` + +It seems not working as the remote hosts do not have /usr/local/bin/ape file. + +Remote nodes can be specified in XML format: cluster-ip-list.xml + +Currently, to create a scenario, the user constructs a shell +script specifying the types of errors to be injected or failures to be simulated, one after another. A sample line in a +scenario file could be as follows: + +``` +java -jar ape.jar -remote cluster-ip-list.xml -F lambda -k lambda + where the -F is a “Fork Bomb” injection, the -k is a “Kill + One Node” command, and the lambda specifies the failure rates. +``` +Users can define lambda parameters by computing Mean +Time Between Failures (MTBF) of a system. MTBF is defined to be the average (or expected) lifetime of a system +and is one of the key decision-making criteria for data center infrastructure systems [1]. Equipment in data centers +is going to fail, and MTBF helps with predicting which systems are the likeliest to fail at any given moment. Based on +previous failure statistics, users can develop an estimate of +MTBF for various equipment failures; however, determining +MTBFs for many software failures is challenging. + +[1] W. Torell and V. Avelar. Performing effective MTBF comparisons for data center infrastructure. +http://www.apcmedia.com/salestools/ASTE-5ZYQF2_R1_EN.pdf. + +Available Commands +------------------ +Here are some common failures in Hadoop environments: +``` +• Data node is killed +• Application Master (AM) is killed +• Application Master is suspended +• Node Manager (NM) is killed +• Node Manager is suspended +• Data node is suspended +• Tasktracker is suspended +• Node panics and restarts +• Node hangs and does not restart +• Random thread within data node is killed +• Random thread within data node is suspended +• Random thread within tasktracker is killed +• Random thread within tasktracker is suspended +• Network becomes slow +• Network is dropping significant numbers of packets +• Network disconnect (simulate cable pull) +• One disk gets VERY slow +• CPU hog consumes x% of CPU cycles +• Mem hog consumes x% of memory +• Corrupt ext3 data block on disk +• Corrupt ext3 metadata block on disk +``` +Command line options: +``` +usage: ape [options] ... + options: + -c,--corrupt-file Corrupt the file given + the address as the first + argument, size as the 2nd + arg, and offset as the + 3rd argument + -C,--corrupt-block Corrupt a random HDFS + block file with a size in + bytes as the 2nd arg and + offset in bytes as the + 3rd argument + -d,--network-disconnect