Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed Typo #242

Open
wants to merge 33 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
9ef6fe2
Update index.md
pooja1pathak Dec 17, 2018
0ab623a
Update quick_start_guide_new.md
pooja1pathak Dec 17, 2018
7e853d6
Update quick_start_guide_old.md
pooja1pathak Dec 17, 2018
98a1caf
Update README.md
pooja1pathak Dec 17, 2018
ff2efee
Update auth_server.md
pooja1pathak Dec 17, 2018
a221c74
Update cygnus.md
pooja1pathak Dec 17, 2018
0886be8
Update diagnosis_procedures.md
pooja1pathak Dec 17, 2018
f8bcb37
Update gui.md
pooja1pathak Dec 17, 2018
86fb381
Update haas_engine_sahara.md
pooja1pathak Dec 17, 2018
714aeac
Update haas_engine_shared_hadoop.md
pooja1pathak Dec 17, 2018
a763948
Update introduction.md
pooja1pathak Dec 17, 2018
7be335f
Update oauth2_hive_auth_provider.md
pooja1pathak Dec 17, 2018
ec64be3
Update oauth2_proxy.md
pooja1pathak Dec 17, 2018
5f5f858
Update sanity_check_procedures.md
pooja1pathak Dec 17, 2018
5d574a9
Update some_words_about_cosmos_and_ecosystem.md
pooja1pathak Dec 17, 2018
edbdcaa
Update storage_cluster.md
pooja1pathak Dec 17, 2018
fdece73
Update tidoop.md
pooja1pathak Dec 17, 2018
ac61ec6
Update README.md
pooja1pathak Dec 17, 2018
eddadbe
Update data_management_and_io.md
pooja1pathak Dec 17, 2018
55ddcf0
Update introduction.md
pooja1pathak Dec 17, 2018
a8da57d
Update request_computing_resources.md
pooja1pathak Dec 17, 2018
0e9ef69
Update using_gui.md
pooja1pathak Dec 17, 2018
ba0a25a
Update using_hadoop_and_ecosystem.md
pooja1pathak Dec 17, 2018
b2af3f5
Update using_oauth2.md
pooja1pathak Dec 17, 2018
91981a6
Update using_tidoop.md
pooja1pathak Dec 17, 2018
f078284
Update README.md
pooja1pathak Dec 17, 2018
6cfb00b
Update README.md
pooja1pathak Dec 17, 2018
8cc799d
Update README.md
pooja1pathak Dec 17, 2018
86916b7
Update README.md
pooja1pathak Dec 17, 2018
925e008
Update README.md
pooja1pathak Dec 17, 2018
5073c89
Update README.md
pooja1pathak Dec 17, 2018
31de825
Update README.md
pooja1pathak Dec 17, 2018
1b7d6a8
Update feeding_sinfonier.md
pooja1pathak Dec 17, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#<a name="top"></a>Cosmos
# <a name="top"></a>Cosmos
[![License Badge](https://img.shields.io/badge/license-AGPL-blue.svg)](https://opensource.org/licenses/AGPL-3.0)
[![Documentation Status](https://readthedocs.org/projects/fiware-cosmos/badge/?version=latest)](http://fiware-cosmos.readthedocs.org/en/latest/?badge=latest)

Expand All @@ -15,7 +15,7 @@ This project is part of [FIWARE](http://fiware.org).

[Top](#top)

##If you want to use Cosmos Global Instance in FIWARE Lab
## If you want to use Cosmos Global Instance in FIWARE Lab
If you are looking for information regarding the specific deployment of Cosmos Global Instance in FIWARE Lab, a HaaS ready to use, please check this documentation:

* [Quick Start Guide](./doc/manuals/quick_start_guide_new.md) for Cosmos users.
Expand All @@ -27,20 +27,20 @@ If you are looking for information regarding the specific deployment of Cosmos G

[Top](#top)

##If you want to deploy and use your own private Hadoop instance
## If you want to deploy and use your own private Hadoop instance
This is the case you don't rely on the Global Instance of Cosmos in FIWARE Lab. In this case, you'll have to install, configure and manage your own Hadoop private instance. The Internet is plenty of documentation that will help you.

[Top](#top)

##If you want to deploy your own public Cosmos instance
## If you want to deploy your own public Cosmos instance
In the (extremly rare) case you are not interested in using the Global Instance of Cosmos or a private instance of Hadoop, but you want to become a Big Data service provider, and you want to base on Cosmos software, you may be interested in the following links:

* [Deployment details](doc/deployment_examples/cosmos/fiware_lab.md) for administrators trying to replicate Cosmos Global Instance in FIWARE Lab.
* In general, you may be insterested in the [Installation and Administration Guide](./doc/manuals/installation_and_administration_manual), also available in [readthedocs](http://fiware-cosmos.readthedocs.io/en/latest/).

[Top](#top)

##Reporting issues and contact information
## Reporting issues and contact information
There are several channels suited for reporting issues and asking for doubts in general. Each one depends on the nature of the question:

* Use [stackoverflow.com](http://stackoverflow.com) for specific questions about this software. Typically, these will be related to installation problems, errors and bugs. Development questions when forking the code are welcome as well. Use the `fiware-cosmos` tag.
Expand Down
20 changes: 10 additions & 10 deletions cosmos-admin/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#<a name="top"></a>Cosmos - Cosmos Admin
# <a name="top"></a>Cosmos - Cosmos Admin

* [What is cosmos-admin](#whatis)
* [Scripts](#scripts)
Expand All @@ -10,7 +10,7 @@
* [Log rotation](#logrotation)
* [Reporting issues and contact information](#contact)

##<a name="whatis"></a>What is cosmos-admin
## <a name="whatis"></a>What is cosmos-admin
cosmos-admin is a set of tools designed to administrate a Cosmos deployment, both for the Sahara-based and shared Hadoop-based flavours.

Available tools are:
Expand All @@ -21,8 +21,8 @@ Available tools are:

[Top](#top)

##<a name="scripts"></a>Scripts
###<a name="datacopier"></a>`data_copier.sh`
## <a name="scripts"></a>Scripts
### <a name="datacopier"></a>`data_copier.sh`
This script has been designed for copying HDFS data from one cluster to another. Please observe the data is copied, not moved, i.e. the source data is never deleted by this script, and this is something up to the source cluster administrator.

The underlying data copying mechanism used by `data_copier.sh` is [WebHDFS](http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html), the RESTful API from Hadoop for HDFS. This API exposes methods for creating directories, renaming files... and, specially, reading and writing files.
Expand All @@ -48,7 +48,7 @@ If for any reason the data copying is interrupted (for instance, the communicati

[Top](#top)

###<a name="getuserstats"></a>`get_user_stats.sh`
### <a name="getuserstats"></a>`get_user_stats.sh`

This script has been designed to get certain user statistics, these ones:

Expand Down Expand Up @@ -79,7 +79,7 @@ At the same time, the system log rotation must be done at the same frequency thi

[Top](#top)

###<a name="warnings"></a>`warnings.sh`
### <a name="warnings"></a>`warnings.sh`
This script has been designed for detecting certain scenarios the Cosmos administrator must be warned about. Specifically:

* When a HDFS space is close to the quota limit. How much close it is depends on a configurable threshold.
Expand All @@ -103,9 +103,9 @@ It is convenient this script is run with a frequency not greater than a day, sin

[Top](#top)

##<a name="osprogramming"></a>OS programming regarding the administrative scripts
## <a name="osprogramming"></a>OS programming regarding the administrative scripts

###<a name="crontab"></a>Crontab
### <a name="crontab"></a>Crontab
`crontab` is the tool used by any Unix-based system to schedule task executions. In this case, we will use this application in order to schedule some of the Cosmos administrative scripts in a periodic fashion.

Scheduling is done through a file that can be edited by typing:
Expand All @@ -127,7 +127,7 @@ Please observe the `get_user_stats.sh` script is executed exactly one minute bef

[Top](#top)

###<a name="logrotation"></a>Log rotation
### <a name="logrotation"></a>Log rotation
Log rotation is usually performed by the `logrotate` application. Since this should be installed and already running by default in all Unix-based machines (check it by running the command `which logrotate`), most probably you will have the following elements in your file system:

* `/etc/logrotate.conf`. Used to configure the application, typically will contain parameters for a weekly rotation of the system logs in general.
Expand Down Expand Up @@ -225,7 +225,7 @@ It is possible the first rotation cycle is not executed at the expected time sin

[Top](#top)

##<a name="contact"></a>Reporting issues and contact information
## <a name="contact"></a>Reporting issues and contact information
There are several channels suited for reporting issues and asking for doubts in general. Each one depends on the nature of the question:

* Use [stackoverflow.com](http://stackoverflow.com) for specific questions about this software. Typically, these will be related to installation problems, errors and bugs. Development questions when forking the code are welcome as well. Use the `fiware-cosmos` tag.
Expand Down
2 changes: 1 addition & 1 deletion cosmos-auth/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#<a name="top"></a>Cosmos Auth server
# <a name="top"></a>Cosmos Auth server
cosmos-auth exposes a RESTful API for [OAuth2](http://oauth.net/2/) tokens generation. These tokens are used in other Cosmos RESTful APIs, such as [WebHDFS](http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html) for authentication/authorization purposes.

In fact, tokens are not really generated by cosmos-auth, but by an Identity Manager (FIWARE's implementation is [Keyrock](http://catalogue.fiware.org/enablers/identity-management-keyrock)) which is accessed by this API. So why not directly accessing the Identity Manager? This is because some sensible information regarding the Cosmos application is needed when requesting a token to the Identity Manager; specifically the `client_id` and `client_secret` generated once the Cosmos application is registered. Thus, in order this information continues being secret, it is necessary this kind of intermediary service.
Expand Down
2 changes: 1 addition & 1 deletion cosmos-gui/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#Cosmos GUI
# Cosmos GUI
This project is part of [FIWARE](http://fiware.org).

[Cosmos](http://catalogue.fiware.org/enablers/bigdata-analysis-cosmos) is the codename for the Reference Implementation of the BigData Generic Enabler of FIWARE. Such a solution is based on the split of storage and computing capabilities:
Expand Down
4 changes: 2 additions & 2 deletions cosmos-hive-auth-provider/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
#<a name="top"></a>OAuth2-based authentication provider for Cosmos
# <a name="top"></a>OAuth2-based authentication provider for Cosmos
cosmos-hive-auth-provider is a custom authentication provider for [Hive](https://hive.apache.org/). Hive natively provides many ways of implementing authentication, e.g. [Kerberos, PAM or LDAP](https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2#SettingUpHiveServer2-Authentication/SecurityConfiguration), but it also allows for configuring custom mechanisms, like this one.

By using cosmos-hive-auth-provider users will be able to authenticate by means or their [OAuth2](https://www.google.es/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0ahUKEwjOj8WZpszKAhXGtBoKHTCvBLMQFgggMAA&url=http%3A%2F%2Foauth.net%2F2%2F&usg=AFQjCNG58oSRksGnoIWIhfWYBB1sg_YGng&sig2=9Sux2Pq4TJwspuoLrFoFrQ) token, generated by a OAuth2 Tokens Generator (a third party) handled by any trusted Identity Manager (for instance, [FIWARE Lab one](https://account.lab.fiware.org/)).

The advantage regarding the way this library has been implemented is that any user-and-password-based Hive client will continue working; simply, the password configuration parameter takes the token value.

Further information can be found in the documentation at [fiware-cosmos.readthedocs.io](http://fiware-cosmos.readthedocs.io/en/latest/).
Further information can be found in the documentation at [fiware-cosmos.readthedocs.io](http://fiware-cosmos.readthedocs.io/en/latest/).
2 changes: 1 addition & 1 deletion cosmos-proxy/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#Custom Http PEP proxy for Cosmos
# Custom Http PEP proxy for Cosmos
cosmos-proxy is a custom Http proxy acting as a Policy Enforcement Point (PEP). This means:

* cosmos-proxy is deployed before any Cosmos Http service aimed to be protected.
Expand Down
2 changes: 1 addition & 1 deletion cosmos-tidoop-api/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#Tidoop REST API
# Tidoop REST API
cosmos-tidoop-api exposes a RESTful API for running MapReduce jobs in a shared Hadoop environment.

Please observe we emphasize in <i>a shared Hadoop environment</i>. This is because shared Hadoops require special management of the data and the analysis processes being run (storage and computation). There are tools like [Oozie](https://oozie.apache.org/) in charge of running MapReduce jobs as well through an API, but they do not take into account the access to the run jobs, their status, results, etc must be controlled. In other words, using Oozie any user may kill a job by knowing its ID; using cosmos-tidoop-api only the owner of the job will be able to.
Expand Down
34 changes: 17 additions & 17 deletions doc/deployment_examples/sinfonier/feeding_sinfonier.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#<a name="top"></a>Sinfonier deployment of Cosmos
# <a name="top"></a>Sinfonier deployment of Cosmos
Content:

* [Introduction](#section1)
Expand All @@ -18,13 +18,13 @@ Content:
* [General procedure step-by-step](#section7)
* [Reporting issues and contact information](#section8)

##<a name="section1"></a>Introduction
## <a name="section1"></a>Introduction
This document describes how Sinfonier can consume historic context information handled by [Orion](https://github.com/telefonicaid/fiware-orion) and stored by [Cygnus](https://github.com/telefonicaid/fiware-cygnus).

The purpose is consuming information with Sinfonier. Orion has context information that could be useful, but we need a way to connect both elements. The deployment of Cygnus and Kafka, for translating and storing the information in a data structure, implements the connection between Orion and Sinfonier.

[Top](#top)
##<a name="section2"></a>Architecture
## <a name="section2"></a>Architecture
The architecture for feeding Sinfonier needs some elements as shown in next image:

![architecture][Architecture]
Expand All @@ -37,7 +37,7 @@ There are two ways to run the architecture:

[Top](#top)

##<a name="section3"></a>Orion contextBroker
## <a name="section3"></a>Orion contextBroker
First of all, [Orion](https://github.com/telefonicaid/fiware-orion/blob/develop/doc/manuals/admin/install.md) must be installed in the system. In addition, Orion needs MongoDB for storage, so must be installed too.

Orion contextBroker must be running in `multiservice` mode with the command:
Expand Down Expand Up @@ -250,8 +250,8 @@ curl -X GET http://localhost:1026/v2/subscriptions

[Top](#top)

##<a name="section4"></a>Cygnus
####<a name=”section4.1></a>Configuration
## <a name="section4"></a>Cygnus
#### <a name=”section4.1></a>Configuration
[Cygnus](https://github.com/telefonicaid/fiware-cygnus) is a connector in charge of persisting Orion context data in certain configured third-party storages, creating a historical view of such data. In other words, Orion only stores the last value regarding an entity's attribute, and if an older value is required then you will have to persist it in other storage, value by value, using Cygnus.

In this architecture Cygnus do the translation between Orion and Kafka. First of all you have to follow a [quick start guide](https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/quick_start_guide.md#installing-cygnus) for install it. Next step is create a properly agent for Kafka. Every agent has to configure three main elements:
Expand Down Expand Up @@ -298,7 +298,7 @@ Some important details:

Running properly all the structure (See [general procedure step-by-step](#section7) for do it properly) and updating some values in our `Entity` you can see how Cygnus persist the information.

####<a name=”section4.2></a>Running
#### <a name=”section4.2></a>Running
Cygnus is run through this command:
```
/path/to/flume/folder/bin/flume-ng agent --conf /path/to/flume/folder/conf -f /path/to/flume/folder/conf/your_kafka_agent.conf -n cygnusagent -Dflume.root.logger=INFO,console
Expand All @@ -318,7 +318,7 @@ As you can see, the information is persisted in a `topic` "Book1_Book" with `att

[Top](#top)

##<a name="section5"></a>Kafka
## <a name="section5"></a>Kafka
[Apache Kafka](http://kafka.apache.org/documentation.html#quickstart) is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.
The use of Kafka for that purpose have two main pieces: Zookeeper and `brokers` (or servers, both names are correct). Kafka is needed for storing the context information handled by the combination of Orion and Cygnus.

Expand All @@ -328,8 +328,8 @@ An important detail is that Zookeeper must be running before `brokers`. Commands

[Top](#top)

###<a name="section5.1"></a>Zookeeper
####<a name=”section5.1.1></a>Configuration
### <a name="section5.1"></a>Zookeeper
#### <a name=”section5.1.1></a>Configuration
Zookeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. The connections to Zookeeper use the port 2181 and provides the link between itself and brokers. Using the API we can create and ask for information about topics, produce some messages and consume, but the most of the actions are done automatically by Cygnus. The only action required for this architecture is to consume, that will carry out Sinfonier.

Zookeeper is configured through the following parameters:
Expand All @@ -344,7 +344,7 @@ Zookeeper configuration must be stored in a file called `zookeeper.properties`

[Top](#top)

####<a name=”section5.1.2></a>Running
#### <a name=”section5.1.2></a>Running
Zookeeper is run through this command:
```
bin/zookeeper-server-start.sh config/zookeeper.properties
Expand All @@ -356,8 +356,8 @@ nohup bin/zookeeper-server-start.sh config/zookeeper.properties &

[Top](#top)

###<a name="section5.2"></a>Brokers
####<a name=”section5.2.1></a>Configuration
### <a name="section5.2"></a>Brokers
#### <a name=”section5.2.1></a>Configuration
`Brokers` are used to distribute the information stored in Kafka and you can use one or several of them. This example use three `brokers` and its configuration is described in this section. `Brokers` are implemented into Zookeeper, each one with its own port, i.e, a `Broker` with port 9092, another broker with port 9093, etc.
A single `Broker` is configured through the following parameters:

Expand All @@ -375,7 +375,7 @@ A “multibroker” configuration can be configured by creating different files,

`Brokers` configurations must be stored in files like `serverx.properties`, being "x" the `broker.id` of each `Broker`.

####<a name=”section5.2.2></a>Running
#### <a name=”section5.2.2></a>Running
A single server is run through this command:
```
bin/kafka-server-start.sh config/server1.properties
Expand All @@ -393,14 +393,14 @@ nohup bin/kafka-server-start.sh config/server1.properties &

[Top](#top)

##<a name="section6"></a>Sinfonier
## <a name="section6"></a>Sinfonier
Finally we reach the last element of our architecture: The consumer of the stored data. Sinfonier works as consumer, asking Kafka for information coming from Orion.

Kafka works as a queue, receiving data from the producers and sending it to the consumers. Regarding this particular architecture, Cygnus works as a producer, while Sinfonier works as a consumer.

[Top](#top)

##<a name="section7"></a>General procedure step-by-step
## <a name="section7"></a>General procedure step-by-step
The following steps will help you to run all the procedure properly. A specific order is required because the architecture need some services running before the others:
1. Orion context broker: First step in order to create the subscriptions and receive the entity updates, that will be redirected to Cygnus. `Mongo` must be running too.
2. Kafka: Zookeeper and `brokers`, and consequently:
Expand All @@ -411,7 +411,7 @@ The following steps will help you to run all the procedure properly. A specific

[Top](#top)

##<a name="section3"></a> Reporting issues and contact information
## <a name="section3"></a> Reporting issues and contact information
There are several channels suited for reporting issues and asking for doubts in general. Each one depends on the nature of the question:

* Use [stackoverflow.com](http://stackoverflow.com) for specific questions about this software. Typically, these will be related to installation problems, errors and bugs. Development questions when forking the code are welcome as well. Use the `fiware-cygnus` tag.
Expand Down
2 changes: 1 addition & 1 deletion doc/manuals/index.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#Welcome to Cosmos
# Welcome to Cosmos
Cosmos is the reference implementation (GEri) of the FIWARE's BigData Analysis Generic Enabler (GE).

Github's [README.md](https://github.com/telefonicaid/fiware-cosmos/blob/master/README.md) and the [Quick Start Guide](../manuals/quick_start_guide_new.md) provide a good documentation summary. The [Installation and Administration Manual](./installation_and_administration_manual/introduction.md) and the [User and Programmer Manual](./user_and_programmer_manual/introduction.md) cover more advanced topics.
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#<a name="top"></a>Cosmos
# <a name="top"></a>Cosmos

* Introduction
* [Intended audience](./introduction.md#section1)
Expand Down
Loading