You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+11-7Lines changed: 11 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,10 +6,11 @@ MarkLogic Content Pump (mlcp) is a command-line tool that provides the fastest w
6
6
* Split and load large, aggregate XML files or delimited text
7
7
* Bulk load billions of triples or quads from RDF files
8
8
* Archive and restore database contents across environments
9
+
* Export data from a database to a file system
9
10
* Copy subsets of data between databases
10
11
* Load documents from HDFS, including Hadoop `SequenceFile`s
11
12
12
-
mlcp can run locally, across many threads on a single machine, or distributedly, across many nodes in a Hadoop cluster.
13
+
You can run mlcp across many threads on a single machine or across many nodes in a Hadoop cluster.
13
14
14
15
The Hadoop Connector is an extension to Hadoop’s MapReduce framework that allows you to easily and efficiently communicate with a MarkLogic database from within a Hadoop job. mlcp uses the Hadoop Connector internally, but it can also be used to build Hadoop MapReduce jobs that interact with MarkLogic, outside of mlcp, such as in a larger Hadoop application. Core features of the Hadoop Connector include:
15
16
@@ -34,9 +35,13 @@ Wiki pages of this project contain useful information when you work on developme
34
35
35
36
-[Wiki Page of marklogic-contentpump](https://github.com/marklogic/marklogic-contentpump/wiki)
36
37
37
-
## Build
38
+
## Required Software
39
+
40
+
-[Required Software for the Hadoop Connector](http://pubs.marklogic.com:8011/9.0/guide/mapreduce/quickstart#id_78738)
41
+
-[Required Software for mlcp](http://pubs.marklogic.com:8011/9.0/guide/mlcp/install#id_44231)
42
+
-[Apache Maven](https://maven.apache.org/) (version >= 3.03) is required to build mlcp and the Hadoop Connector.
38
43
39
-
mlcp and the Hadoop Connector use [Apache Maven](https://maven.apache.org/) (version >= 3.03) to build.
44
+
## Build
40
45
41
46
mlcp and Hadoop Connector can be built together. Steps to build:
The build writes to the respective **deliverable** directories under the top-level `./mlcp/` and `./mapreduce/` directories.
50
55
51
-
Alternatively, you can build mlcp and the Hadoop Connector independently from each component’s root directory (i.e. `./mlcp/` and `./mapreduce/`) with above command. *Note that mlcp depends on the Hadoop Connector.* So successful build of Hadoop Connector is required to build mlcp.
56
+
Alternatively, you can build mlcp and the Hadoop Connector independently from each component’s root directory (i.e. `./mlcp/` and `./mapreduce/`) with the above command. *Note that mlcp depends on the Hadoop Connector*, so a successful build of the Hadoop Connector is required to build mlcp.
52
57
53
58
For information on contributing to this project see [CONTRIBUTING.md](https://github.com/marklogic/marklogic-contentpump/blob/8.0-master/CONTRIBUTING.md). For information on working on development of this project see [project wiki page](https://github.com/marklogic/marklogic-contentpump/wiki).
54
59
55
-
56
60
## Tests
57
61
58
-
The unit tests included in this repository are designed to provide illustrative examples of the APIs and to sanity check external contributions. MarkLogic Engineering runs a more comprehensive set of unit, integration, and performance tests internally. To run unit tests, under`marklogic-contentpump/` root directory:
62
+
The unit tests included in this repository are designed to provide illustrative examples of the APIs and to sanity check external contributions. MarkLogic Engineering runs a more comprehensive set of unit, integration, and performance tests internally. To run the unit tests, execute the following command from the`marklogic-contentpump/` root directory:
59
63
60
64
```bash
61
65
$ mvn test
@@ -65,7 +69,7 @@ For detailed information about running unit tests, see [Guideline to Run Tests](
65
69
66
70
## Have a question? Need help?
67
71
68
-
If you have questions about how to use mlcp or the Hadoop Connector, you can ask on [StackOverflow](http://stackoverflow.com/questions/tagged/mlcp). Remember to tag the question with [**mlcp** and **marklogic**](http://stackoverflow.com/questions/tagged/mlcp+marklogic). If you find a bug or would like to propose a new capability [file a GitHub issue](https://github.com/marklogic/marklogic-contentpump/issues/new).
72
+
If you have questions about mlcp or the Hadoop Connector, ask on [StackOverflow](http://stackoverflow.com/questions/tagged/mlcp). Tag your question with [**mlcp** and **marklogic**](http://stackoverflow.com/questions/tagged/mlcp+marklogic). If you find a bug or would like to propose a new capability,[file a GitHub issue](https://github.com/marklogic/marklogic-contentpump/issues/new).
0 commit comments