feat: initial commit

Author: iBotPeaches

.gitignore

@@ -0,0 +1,3 @@
+.DS_Store
+.idea
+

DOCKER_TIPS.md

@@ -0,0 +1,41 @@
+## Typical Docker Commands
+    ### Start all environments services
+    > docker-compose up --build
+    
+    ### Restart a specific container
+    > docker restart [service name]
+    
+    ### List all containers
+    > docker ps -a
+    
+    ### List running containers
+    > docker ps
+    
+    ### See used space
+    > docker system df
+    
+    ### Remove un-used data
+    > docker system prune
+
+## Passwords
+    ### PostgreSQL
+    > username: postgres_user
+    > password: postgres_pass
+    
+    ### MySql
+    > username: mysql_user
+    > password: mysql_pass
+    
+    ### MariaDB
+    > username: mariadb_user
+    > password: mariadb_pass
+
+
+## Running Xdebug with PHPStorm alongside Docker.
+Any logic that needs to speak to the host machine will need to point to the IP of `10.254.254.254`.
+
+For Xdebug to work through PHPStorm first execute:
+
+```
+sudo ifconfig lo0 alias 10.254.254.254
+```

README.md

@@ -0,0 +1,46 @@
+# Sourcetoad Internal Development Tools (SIDT)
+
+Sourcetoad Internal Development Tools includes a growing collection of tutorials
+for managing IDEs, hooking up tools with Docker and any combination of those 
+things.
+
+Additionally, this repository holds our data source services which is 
+a collection of databases and other services that projects will more than likely 
+need. It also provides a proxy setup via nginx-proxy so that multiple projects 
+can be run at the same time without the need for different ports.
+
+This means docker configurations for each project only needs to worry about the 
+the language and other tools it may need, as the databases and other services are
+standardized in this project. Since most cloud providers provide managed versions
+of these services, you likely would not use a docker version of them in production.
+
+## Docker
+
+### devop-tools (data source services / domain tools)
+1. Execute `git clone [email protected]:sourcetoad/devop-tools.git`
+2. `cd devop-tools`
+3. `cd ./docker/data-source-services/`
+4. Execute `./network-creation.sh`
+- If this fails, then manual configuration is necessary before compilation. Perform the following:
+   1. `docker network create nginx-proxy`
+   2. `docker network create st-mariadb-101`
+   3. `docker network create st-mysql-56`
+   4. `docker network create st-postgres-95`
+   5. `docker network create st-redis-32`
+5. Execute `docker-compose up --build`
+5. Wait for the terminal to complete executing.
+6. You now have Sourcetoad data sources running and logging to the console.
+
+## Examples
+Inside the `examples` folder you will find example Docker configurations for
+popular frameworks like Laravel and Yii2.
+
+The examples make the assumption that any domain used for local development is 
+already added to the `/etc/hosts` file. You can edit to add `127.0.0.1 domain.docker`.
+
+## Docs
+ * [Setting up D4M-NFS](docs/d4m-nfs/README.md)
+ * [Setting up Nginx-Proxy](docs/nginx-proxy/README.md)
+ * [Setting up PHP Testing in PHPStorm](docs/phpstorm-docker/README.md)
+ * [Setting up Python Debugging in Pycharm](docs/pycharm-debugging/README.md)
+ * [Building a new Upsource Project](docs/upsource/README.md)

docker/data-source-services/docker-compose.yml

@@ -0,0 +1,80 @@
+# Last two digits of port number are the MAJOR.MINOR of the data source
+# Everything ahead of the last two digits are standard for the given service
+#
+# Network name is st-{service name}-{service MAJOR.MINOR}
+version: '2'
+services:
+  nginxproxy:
+# Enable nginx debug and increase output verbosity
+#    command: [nginx-debug, '-g', 'daemon off;']
+    image: jwilder/nginx-proxy
+    container_name: sourcetoad_nginx_proxy
+    ports:
+      - "80:80"
+    volumes:
+      - /var/run/docker.sock:/tmp/docker.sock:ro
+      - ./proxy_increase.conf:/etc/nginx/proxy.conf
+    networks:
+     - nginx-proxy
+  postgres95:
+    image: postgres:9.5
+    container_name: sourcetoad_postgres95
+    ports:
+      - "5495:5432"
+    environment:
+      - POSTGRES_USER=postgres_user
+      - POSTGRES_PASSWORD=postgres_pass
+    networks:
+      - st-postgres-95
+    command: "postgres
+                -c logging_collector='on'
+                -c log_directory='/var/log/postgresql'
+                -c log_filename='%Y-%m-%d.log'
+                -c log_line_prefix='%t %v '
+                -c log_statement='all'"
+  mysql56:
+    image: mysql:5.6
+    container_name: sourcetoad_mysql56
+    ports:
+      - "3356:3306"
+    environment:
+      - MYSQL_ROOT_PASSWORD=root
+      - MYSQL_USER=mysql_user
+      - MYSQL_PASSWORD=mysql_pass
+    networks:
+      - st-mysql-56
+  mariadb101:
+    image: mariadb:10.1
+    container_name: sourcetoad_mariadb101
+    ports:
+      - "33101:3306"
+    environment:
+      - MYSQL_ROOT_PASSWORD=root
+      - MYSQL_USER=mariadb_user
+      - MYSQL_PASSWORD=mariadb_pass
+    networks:
+      - st-mariadb-101
+  redis32:
+    image: redis:3.2-alpine
+    command: redis-server --appendonly yes
+    container_name: sourcetoad_redis32
+    ports:
+      - "6332:6379"
+    networks:
+      - st-redis-32
+networks:
+    nginx-proxy:
+        external:
+            name: nginx-proxy
+    st-postgres-95:
+        external:
+            name: st-postgres-95
+    st-mysql-56:
+        external:
+            name: st-mysql-56
+    st-mariadb-101:
+        external:
+            name: st-mariadb-101
+    st-redis-32:
+        external:
+            name: st-redis-32

docker/data-source-services/network-creation.sh

@@ -0,0 +1,6 @@
+echo "Creating container networks:\n"
+docker network create nginx-proxy
+docker network create st-mariadb-101
+docker network create st-mysql-56
+docker network create st-postgres-95
+docker network create st-redis-32

docker/data-source-services/proxy_increase.conf

@@ -0,0 +1,22 @@
+# nginx-proxy uses defaults for client_max_body_size, which is low for some projects
+client_max_body_size 1000m;
+
+# nginx-proxy uses defaults for proxy timeouts, which is low if using remote debugging
+proxy_connect_timeout 5;
+proxy_send_timeout 3600;
+proxy_read_timeout 3600;
+
+# nginx-proxy sets the values below by default only IF a proxy change is not defined, therefore, we duplicate the default values
+proxy_http_version 1.1;
+proxy_buffering off;
+proxy_set_header Host $http_host;
+proxy_set_header Upgrade $http_upgrade;
+proxy_set_header Connection $proxy_connection;
+proxy_set_header X-Real-IP $remote_addr;
+proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
+proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
+proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
+
+# Mitigate httpoxy attack (see README for details)
+proxy_set_header Proxy "";

docs/d4m-nfs/README.md

@@ -0,0 +1,30 @@
+# D4M-NFS on Mac OSX
+
+*Warning: Currently D4M-NFS is broken after the update to 17.12.0-ce-mac64. The issue can be followed here: https://github.com/IFSight/d4m-nfs/issues/55*
+
+In order to speed up Docker instances on Mac OSX, D4M-NFS is a great solution that will increase file read/write performance up to 6X times.
+
+### Steps for success!
+
+1. Shut down any docker containers `docker stop $(docker ps -a -q)` otherwise you could get a "loop of death" which isn't fun from what I've read.
+2. Quit the docker daemon/application.
+3. Kill d4m if you have an old, non-working version running. You can determine this by running `screen -r d4m`
+4. `cd ~/`
+5. `git clone [email protected]:IFSight/d4m-nfs.git ~/d4m-nfs`
+6. `cd ~/d4m-nfs/etc`
+7. `touch d4m-nfs-mounts.txt`
+8. Add the text `/Users/yourusername:/Users/yourusername:0:0`
+    - Do not add /Volumes here, this should be the only thing added for this file.
+9. Go into docker and remove everything else except /tmp.  
+![docker](images/docker-file-sharing.png)
+    - If you add other directories, this will not work as it will create conflicts with the NFS systems.
+10. `cd ~/d4m-nfs`
+11. `/bin/bash d4m-nfs.sh`
+12. `sudo vim /etc/exports`
+    - Remove any other non-commented lines except for `"/Users/yourusername" -alldirs -mapall=0:0 localhost`
+    - If you see anything other than this you were not running with bash.
+11. Start up ya dockers, rejoice!
+
+#### Mount Points
+
+I have not had success with adding code to a /mnt point and getting D4M-NFS to work. Please update the documentation if you have a solution.

docs/d4m-nfs/images/docker-file-sharing.png

@@ -0,0 +1,28 @@
+# nginx-proxy
+
+The data source services includes an application called nginx-proxy. This allows you to run multiple projects at the same time on different virtual hosts without port conflicts. You do not need to expose the webserver to the host machine through a port map.
+
+In order for nginx-proxy to operate, you must follow the following steps.
+
+## Setup hosts file
+1. Choose a virtual host for your project. RFC-6761 recommends against using a .dev TLD, and instead using a .localhost TLD. Therefore, your domain might be mysite.localhost
+2. Edit your hosts file and add this domain to your loopback address (127.0.0.1) using your preferred text editor. *make sure to use sudo*
+```
+sudo nano /etc/hosts
+
+add:
+127.0.0.1 mysite.localhost
+```
+
+## Setup docker-compose for project
+1. In order for nginx-proxy to detect and proxy to the correct virtual host, you must first identify the container that is running the "web server." In a php project, this would be the nginx container, for example.
+2. You will then add an environment variable of `VIRTUAL_HOST=mysite.localhost` to this service, place that service on the nginx-proxy network, and import that nginx-proxy network to the project. See the below snippet:
+
+```
+version: '2'
+services:
+    [APP_NAME]_nginx:
+        image: nginx:latest
+        environment:
+            - VIRTUAL_HOST=mysite.localhost
+```

docs/nginx-proxy/README.md

@@ -0,0 +1,128 @@
+# PHPStorm & Docker
+
+The goal for this tutorial is to help containerize an application
+to leverage the tools within PHPStorm (Test Suite, Coverage, etc).
+
+To do this the following things need to happen:
+
+1) Setup Docker for Mac in PHPStorm.
+2) Setup Language (PHP in this example) to use a remote interpreter.
+3) Fix the paths to match our standard (/code for project code)
+4) Setup Tests (PHPUnit or Codeception) to use a special docker-compose file.
+
+We are assuming the following things:
+
+1) Docker compose file exists for the project.
+2) The project is already running via (docker-compose up --build)
+3) Some small knowledge of Docker
+
+### Setup Docker for Mac in PHPStorm
+
+Head to Preferences and go to `Build, Execution, Deployment -> Docker`. Create a
+an item called "Docker" using the daemon option "Docker for Mac".
+
+![](./assets/docker-phpstorm.png)
+
+### Creating an isolated docker-compose for testing.
+
+In order for PHPStorm to run tests without affecting containers. You need to 
+create a new container that matches whatever container your code works on.
+
+For example, here a random PHP project `docker-compose-phpstorm.yml` file.
+
+```
+version: '2'
+services:
+    project-test:
+        container_name: sourcetoad_project_test
+        build: .
+        volumes:
+            - ../:/code:delegated
+        networks:
+            - project-internal
+            - st-mariadb-101
+            - st-redis-32
+networks:
+    project-internal:
+    st-redis-32:
+        external:
+            name: st-redis-32
+    st-mariadb-101:
+        external:
+            name: st-mariadb-101
+```
+
+As you can see, it is a subset of the compose file. It only includes a clone of 
+the PHP container, keeping intact the networks, mounts and env file. This allows
+the new container to have access to databases and files which may be needed in
+the test suite. You can use multiple compose files if you need one for an 
+acceptance test container.
+
+If your project requires multiple containers in the testing aspect. Recommend 
+creating descriptive container names, like `project-codeception` so you could 
+additionally have `project-behat` if needed.
+
+### Setup Docker Container for PHP Language
+
+Head to Preferences and go to `Languages & Frameworks > PHP`. From here we are 
+going to click the "..." next to the CLI Interpreter option in order to create a
+new "Docker Compose" interpreter. Using the file we created above, we will 
+select that one. Once selected, you need to select the container to use. This 
+will more than likely be auto-selected for you. 
+
+![](./assets/docker-findphp.png)
+
+### Setup Docker Container for PHP Test Framework
+
+Head to Preferences and go to `Languages & Framework > PHP > Test Frameworks`. 
+Once here, click the "+" icon in top right to create a new test environment. You
+will more than likely select one of the following:
+
+ * PHPUnit by Remote Interpreter
+ * Codeception by Remote Interpreter
+ * Behat by Remote Interpreter
+ * PHPSpec by Remote Interpreter
+ 
+Once you select one, you will need to select the container to run this tool at.
+You already created the container and loaded it in the previous steps. So just
+select the container that works. Once loaded, you need to put in the file path 
+to that binary. In the screenshot below, we are using Codeception so looking
+for `/code/vendor/bin/codecept`. The `/code` is what we use to mount our code 
+into the container. This pattern may be different depending on lots of factors.
+
+Finally, for this step you need to make sure you specify the configuration file 
+for the test framework. The default location may be wrong, in the photo below I
+gave the direct path to the configuration like: `/code/codeception.yml`.
+ 
+![](./assets/docker-codecept.png)
+ 
+### Setup Test Case to run in IDE
+ 
+Head "Run -> Edit Configurations". Now you need to select the test framework 
+that matches the test framework you attached into the container. Since, we used
+Codeception, we will continue to use Codeception for this demo.
+ 
+Our goal is to select the "Defined in the Configuration File". No matter the 
+tool, if you configured it correctly, we can just use those values and inherit.
+ 
+This demo has made an additional change and requested Codeception to only run 
+the "unit" test suite.
+ 
+![](./assets/docker-unittest.png)
+
+### Run the Test Suite
+
+Now we should be able to the launch the test suite. Our docker-compose file will
+execute, spawning a new "test" container based off our main language container.
+
+This will then proceed to execute the test framework we specified, collecting 
+the results and parsing them into the UI. 
+
+This can be shown below: 
+
+![](./assets/docker-testpass.png)
+
+With that you have successfully hooked the test suites, while keeping Docker 
+inside PHPStorm.
+
+