Skip to content

Commit a0b48c4

Browse files
rsdoielrsdoiel
committed
Initial Setup
0 parents  commit a0b48c4

37 files changed

+2804
-0
lines changed

‎.gitignore

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Compiled Object files, Static and Dynamic libs (Shared Objects)
2+
*.o
3+
*.a
4+
*.so
5+
6+
# Folders
7+
_obj
8+
_test
9+
10+
# Architecture specific extensions/prefixes
11+
*.[568vq]
12+
[568vq].out
13+
14+
*.cgo1.go
15+
*.cgo2.c
16+
_cgo_defun.c
17+
_cgo_gotypes.go
18+
_cgo_export.*
19+
20+
_testmain.go
21+
22+
*.exe
23+
*.test
24+
*.prof
25+
26+
#
27+
# Project specific files
28+
#
29+
*~
30+
*.swp
31+
*.zip
32+
dist/*
33+
bin/*
34+

‎INSTALL.md

+80
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
2+
# Installation
3+
4+
## Compiled version
5+
6+
*datatools* is a collection of command line programs run from a shell like Bash (or Powershell in Windows). If you download the repository a compiled version is in the dist directory. The compiled VERSION_NO matching your computer type and operating system can be copied to a bin directory in your PATH.
7+
8+
Compiled versions are available for Mac OS X (amd64 processor), Linux (amd64), Windows (amd64) and Rapsberry Pi (both ARM6 and ARM7)
9+
10+
### Mac OS X
11+
12+
1. Download **datatools-VERSION_NO-release.zip** from [https://github.com/caltechlibrary/datatools/releases/latest](https://github.com/caltechlibrary/datatools/releases/latest)
13+
2. Open a finder window, find and unzip **datatools-VERSION_NO-release.zip**
14+
3. Look in the unziped folder and find *dist/macosx-amd64/*
15+
4. Drag (or copy) *jsoncols*, *jsonrange*, etc. to a "bin" directory in your path
16+
5. Open and "Terminal" and run `jsoncols -h` to confirm you were successful
17+
18+
### Windows
19+
20+
1. Download **datatools-VERSION_NO-release.zip** from [https://github.com/caltechlibrary/datatools/releases/latest](https://github.com/caltechlibrary/datatools/releases/latest)
21+
2. Open the file manager find and unzip **datatools-VERSION_NO-release.zip**
22+
3. Look in the unziped folder and find *dist/windows-amd64/*
23+
4. Drag (or copy) *jsoncols.exe*, *jsonrange.exe*, etc. to a "bin" directory in your path
24+
5. Open Bash and and run `jsoncols -h` to confirm you were successful
25+
26+
### Linux
27+
28+
1. Download **datatools-VERSION_NO-release.zip** from [https://github.com/caltechlibrary/datatools/releases/latest](https://github.com/caltechlibrary/datatools/releases/latest)
29+
2. Find and unzip **datatools-VERSION_NO-release.zip**
30+
3. In the unziped directory and find for *dist/linux-amd64/*
31+
4. Copy *jsoncols*, *jsonrange*, etc. to a "bin" directory (e.g. cp ~/Downloads/datatools-VERSION_NO-release/dist/linux-amd64/\* ~/bin/)
32+
5. From the shell prompt run `jsoncols -h` to confirm you were successful
33+
34+
### Raspberry Pi
35+
36+
If you are using a Raspberry Pi 2 or later use the ARM7 VERSION_NO, ARM6 is only for the first generaiton Raspberry Pi.
37+
38+
1. Download **datatools-VERSION_NO-release.zip** from [https://github.com/caltechlibrary/datatools/releases/latest](https://github.com/caltechlibrary/datatools/releases/latest)
39+
2. Find and unzip **datatools-VERSION_NO-release.zip**
40+
3. In the unziped directory and find for *dist/raspberrypi-arm7/*
41+
4. Copy *jsoncols*, *jsonrange*, etc. to a "bin" directory (e.g. cp ~/Downloads/datatools-VERSION_NO-release/dist/raspberrypi-arm7/\* ~/bin/)
42+
+ if you are using an original Raspberry Pi you should copy the ARM6 version instead
43+
5. From the shell prompt run `jsoncols -h` to confirm you were successful
44+
45+
46+
## Compiling from source
47+
48+
If you have go v1.7.4 or better installed then should be able to "go get" to install all the **datatools** utilities and
49+
50+
```
51+
go get -u github.com/caltechlibrary/datatools/...
52+
```
53+
54+
Or for Windows 10 Powershell (assumes the Windows versions of Go and Git are previously installed)
55+
56+
57+
```powershell
58+
$Env:GOPATH = "$HOME"
59+
go get -u github.com/caltechlibrary/datatools/...
60+
```
61+
62+
or to install from source on Linux/Unix systems
63+
64+
```bash
65+
git clone https://github.com/caltechlibrary/datatools src/github.com/caltechlibrary/datatools
66+
cd src/github.com/caltechlibrary/datatools
67+
make
68+
make test
69+
make install
70+
```
71+
72+
Or for Windows 10 Powershell
73+
74+
```powershell
75+
$Env:GOBIN = "$HOME\bin"
76+
git clone https://github.com/caltechlibrary/datatools src/github.com/caltechlibrary/datatools
77+
cd src\github.com\caltechlibrary\datatools
78+
go install cmds\jsoncols\filefile.go
79+
```
80+

‎LICENSE

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
2+
Copyright (c) 2017, Caltech
3+
All rights not granted herein are expressly reserved by Caltech.
4+
5+
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
6+
7+
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
8+
9+
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
10+
11+
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
12+
13+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎Makefile

+135
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
#
2+
# Simple Makefile
3+
#
4+
PROJECT = datatools
5+
6+
VERSION = $(shell grep -m1 'Version = ' $(PROJECT).go | cut -d\" -f 2)
7+
8+
BRANCH = $(shell git branch | grep '* ' | cut -d\ -f 2)
9+
10+
build: bin/csvcols bin/csvjoin bin/jsoncols bin/jsonrange bin/xlsx2json bin/xlsx2csv bin/csv2mdtable bin/csv2xlsx
11+
12+
bin/csvcols: datatools.go cmds/csvcols/csvcols.go
13+
go build -o bin/csvcols cmds/csvcols/csvcols.go
14+
15+
bin/csvjoin: datatools.go cmds/csvjoin/csvjoin.go
16+
go build -o bin/csvjoin cmds/csvjoin/csvjoin.go
17+
18+
bin/jsoncols: datatools.go cmds/jsoncols/jsoncols.go
19+
go build -o bin/jsoncols cmds/jsoncols/jsoncols.go
20+
21+
bin/jsonrange: datatools.go cmds/jsonrange/jsonrange.go
22+
go build -o bin/jsonrange cmds/jsonrange/jsonrange.go
23+
24+
bin/xlsx2json: datatools.go cmds/xlsx2json/xlsx2json.go
25+
go build -o bin/xlsx2json cmds/xlsx2json/xlsx2json.go
26+
27+
bin/xlsx2csv: datatools.go cmds/xlsx2csv/xlsx2csv.go
28+
go build -o bin/xlsx2csv cmds/xlsx2csv/xlsx2csv.go
29+
30+
bin/csv2mdtable: datatools.go cmds/csv2mdtable/csv2mdtable.go
31+
go build -o bin/csv2mdtable cmds/csv2mdtable/csv2mdtable.go
32+
33+
bin/csv2xlsx: datatools.go cmds/csv2xlsx/csv2xlsx.go
34+
go build -o bin/csv2xlsx cmds/csv2xlsx/csv2xlsx.go
35+
36+
website:
37+
./mk-website.bash
38+
39+
status:
40+
git status
41+
42+
save:
43+
git commit -am "Quick Save"
44+
git push origin $(BRANCH)
45+
46+
refresh:
47+
git fetch origin
48+
git pull origin $(BRANCH)
49+
50+
publish:
51+
./mk-website.bash
52+
./publish.bash
53+
54+
clean:
55+
if [ -f index.html ]; then /bin/rm *.html;fi
56+
if [ -d bin ]; then /bin/rm -fR bin; fi
57+
if [ -d dist ]; then /bin/rm -fR dist; fi
58+
if [ -f $(PROJECT)-$(VERSION)-release.zip ]; then rm -f $(PROJECT)-$(VERSION)-release.zip; fi
59+
60+
install:
61+
env GOBIN=$(HOME)/bin go install cmds/csvcols/csvcols.go
62+
env GOBIN=$(HOME)/bin go install cmds/csvjoin/csvjoin.go
63+
env GOBIN=$(HOME)/bin go install cmds/jsoncols/jsoncols.go
64+
env GOBIN=$(HOME)/bin go install cmds/jsonrange/jsonrange.go
65+
env GOBIN=$(HOME)/bin go install cmds/xlsx2json/xlsx2json.go
66+
env GOBIN=$(HOME)/bin go install cmds/xlsx2csv/xlsx2csv.go
67+
env GOBIN=$(HOME)/bin go install cmds/csv2mdtable/csv2mdtable.go
68+
env GOBIN=$(HOME)/bin go install cmds/csv2xlsx/csv2xlsx.go
69+
70+
dist/linux-amd64:
71+
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/csvcols cmds/csvcols/csvcols.go
72+
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/csvjoin cmds/csvjoin/csvjoin.go
73+
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/jsoncols cmds/jsoncols/jsoncols.go
74+
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/jsonrange cmds/jsonrange/jsonrange.go
75+
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/xlsx2json cmds/xlsx2json/xlsx2json.go
76+
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/xlsx2csv cmds/xlsx2csv/xlsx2csv.go
77+
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/csv2mdtable cmds/csv2mdtable/csv2mdtable.go
78+
env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/csv2xlsx cmds/csv2xlsx/csv2xlsx.go
79+
80+
dist/macosx-amd64:
81+
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/csvcols cmds/csvcols/csvcols.go
82+
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/csvjoin cmds/csvjoin/csvjoin.go
83+
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/jsoncols cmds/jsoncols/jsoncols.go
84+
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/jsonrange cmds/jsonrange/jsonrange.go
85+
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/xlsx2json cmds/xlsx2json/xlsx2json.go
86+
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/xlsx2csv cmds/xlsx2csv/xlsx2csv.go
87+
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/csv2mdtable cmds/csv2mdtable/csv2mdtable.go
88+
env CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o dist/macosx-amd64/csv2xlsx cmds/csv2xlsx/csv2xlsx.go
89+
90+
dist/windows-amd64:
91+
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/csvcols.exe cmds/csvcols/csvcols.go
92+
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/csvjoin.exe cmds/csvjoin/csvjoin.go
93+
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/jsoncols.exe cmds/jsoncols/jsoncols.go
94+
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/jsonrange.exe cmds/jsonrange/jsonrange.go
95+
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/xlsx2json.exe cmds/xlsx2json/xlsx2json.go
96+
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/xlsx2csv.exe cmds/xlsx2csv/xlsx2csv.go
97+
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/csv2mdtable.exe cmds/csv2mdtable/csv2mdtable.go
98+
env CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/csv2xlsx.exe cmds/csv2xlsx/csv2xlsx.go
99+
100+
dist/raspbian-arm7:
101+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/csvcols cmds/csvcols/csvcols.go
102+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/csvjoin cmds/csvjoin/csvjoin.go
103+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/jsoncols cmds/jsoncols/jsoncols.go
104+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/jsonrange cmds/jsonrange/jsonrange.go
105+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/xlsx2json cmds/xlsx2json/xlsx2json.go
106+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/xlsx2csv cmds/xlsx2csv/xlsx2csv.go
107+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/csv2mdtable cmds/csv2mdtable/csv2mdtable.go
108+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -o dist/raspbian-arm7/csv2xlsx cmds/csv2xlsx/csv2xlsx.go
109+
110+
dist/raspbian-arm6:
111+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/csvcols cmds/csvcols/csvcols.go
112+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/csvjoin cmds/csvjoin/csvjoin.go
113+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/jsoncols cmds/jsoncols/jsoncols.go
114+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/jsonrange cmds/jsonrange/jsonrange.go
115+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/xlsx2json cmds/xlsx2json/xlsx2json.go
116+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/xlsx2csv cmds/xlsx2csv/xlsx2csv.go
117+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/csv2mdtable cmds/csv2mdtable/csv2mdtable.go
118+
env CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o dist/raspbian-arm6/csv2xlsx cmds/csv2xlsx/csv2xlsx.go
119+
120+
121+
release: dist/linux-amd64 dist/macosx-amd64 dist/windows-amd64 dist/raspbian-arm7 dist/raspbian-arm6
122+
mkdir -p dist
123+
cp -v README.md dist/
124+
cp -v LICENSE dist/
125+
cp -v INSTALL.md dist/
126+
cp -v csvcols.md dist/
127+
cp -v csvjoin.md dist/
128+
cp -v jsoncols.md dist/
129+
cp -v jsonrange.md dist/
130+
cp -v xlsx2json.md dist/
131+
cp -v xlsx2csv.md dist/
132+
cp -v csv2mdtable.md dist/
133+
cp -v csv2xlsx.md dist/
134+
zip -r $(PROJECT)-$(VERSION)-release.zip dist/
135+

‎README.md

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
2+
# datatools
3+
4+
Various utilities for simplifying JSON, Excel Workbook and CSV data work on the command line.
5+
6+
+ [csvcols](csvcols.html) - a tool for formatting command line arguments intoa CSV row or filtering CSV rows for specific columns
7+
+ [csvjoin](csvjoin.html) - a tool to join to CSV files on common values in designated columns, writes combined CSV rows to stdout
8+
+ [csv2mdtable](csv2mdtable.html) - a tool to render CSV as a Github Flavored Markdown table
9+
+ [jsoncols](jsoncols.html) - a tool for exploring and extracting JSON values into columns
10+
+ [jsonrange](jsonrange.html) - a tool for iterating for JSON maps and arrays
11+
+ [xlsx2json](xlsx2json.html) - a tool for converting Excel Workbooks to JSON files
12+
+ [xlsx2csv](xlsx2csv.html) - a tool for converting Excel Workbooks sheets to a CSV file(s)
13+
14+
Compiled versions are provided for Linux (amd64), Mac OS X (amd64),
15+
Windows 10 (amd64) and Raspbian (ARM6, ARM7). See https://github.com/caltechlibrary/datatools.
16+
17+
Use the utilities try "-help" option for a full list of options.
18+
19+
20+
## Installation
21+
22+
_datatools_ is go get-able.
23+
24+
```
25+
go get github.com/caltechlibrary/datatools/...
26+
```
27+
28+
Or see [INSTALL.md](install.html) for details for installing
29+
compiled versions of the programs.
30+
31+

‎TODO.md

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
2+
# Someday, Maybe
3+
4+
+ csv2json would convert CSV content to a 2d-JSON array
5+
+ E.g. `cat file1.csv | csv2json`
6+
+ csv2xlsx would take CSV content piped from stdin and write it do a sheet in an Excel file (creating the Excel workbook if needed)
7+
+ E.g. `cat file1.csv | csv2xlsx MyWorkbook.xlsx "Sheet from file1.csv"`
8+
+ csvfind would accept CSV input from stdin and output rows with matching column values
9+
+ E.g. `cat file1.csv | csvfind -column=3 "Book"`
10+
+ json2csv would convert a 2d JSON array to CSV output, it would comvert a JSON object/map to a column of keys next to a column of values
11+
+ E.g. `cat data.json | json2csv`
12+

‎assets/liblogo.gif

4.33 KB
Loading

‎assets/orangebooks-logo-icon.png

1.48 KB
Loading

‎assets/orangebooks-logo.png

10 KB
Loading

0 commit comments

Comments
 (0)