Implement initial working stack/service

Signed-off-by: Adam Crowder <[email protected]>

.editorconfig

@@ -0,0 +1,2 @@
+[*]
+insert_final_newline = true

.eslintignore

@@ -0,0 +1,4 @@
+# Built TS
+dist/
+
+**/*.js

.gitignore

@@ -0,0 +1,42 @@
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+lerna-debug.log*
+
+# Coverage directories
+coverage
+*.lcov
+.nyc_output
+
+# Dependencies
+node_modules/
+
+# Cache directories
+*.tsbuildinfo
+.npm
+.eslintcache
+.cache
+
+# Optional REPL history
+.node_repl_history
+
+# ide
+.idea
+*.swp
+.vscode
+
+# OS Files
+.DS_Store
+desktop.ini
+.direnv
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out
+
+# Built code
+*.js
+*.d.ts

.npmignore

@@ -0,0 +1,6 @@
+*.ts
+!*.d.ts
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out

README.md

@@ -0,0 +1,105 @@
+# gitea-to-s3
+
+This repository provides a native aws stack using lambda, apigateway, and s3 which can
+receive webhooks from a configured gitea instance, and then subsequently fetch and
+upload the code from the related webhook event to S3, for use with consumption in codebuild and codepipeline.
+
+Feel free to fork and contribute, or raise issues if desired.
+
+## Setup
+
+### CDK Environment
+
+You will first need to make sure that you have [nodejs](https://nodejs.org/en/download/) installed with npm.
+
+This also assumes you already have installed and configured the [aws cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
+with your desired aws account.
+
+1. Download the code repository: `git clone https://github.com/cheeseandcereal/gitea-to-s3.git && cd gitea-to-s3`
+1. Install the dependencies: `npm install`
+1. Ensure that you have cdk installed globally: `npm install -g aws-cdk`
+1. Ensure that you have the latest CDK bootstrap setup: `cdk bootstrap`
+
+### Gitea Setup
+
+#### Application Token
+
+You will need an application token to provide this aws service so it can access your gitea instance API.
+
+1. While logged into your gitea instance, click on your profile in the top right and select 'Settings'
+1. Select the 'Applications' tab, then fill in the 'Token Name' text box with anything you want
+1. Click the 'Generate Token' button, then save the token string that is generated
+
+#### API URL
+
+You will need the api url for your gitea instance. This should just your gitea host followed by `api/v1`
+
+I.e: `https://my.gitea.com/api/v1`
+
+Determine this url and save it for the next step
+
+### Deploy the Stack
+
+The aws application stack can now be deployed
+
+Run this command, replacing the valuse for `giteaApiUrl` and `giteaApiToken` as appropriate
+with values from the previous Gitea Setup steps.
+
+`cdk --context giteaApiUrl=https://my.gitea.com/api/v1 --context giteaApiToken=abc123 deploy`
+
+After this command runs, it will print out 2 outputs: one for the name of the s3 bucket, and another for the endpoint URL,
+make sure to save these.
+
+### Setup Code Repos To Send To The Stack
+
+Now that the application stack is running, you can configure the repos in your gitea instance that you want to setup
+for ci/cd. Note you must be an admin with permissions to modify the repo settings (for the desired repos) in order to do this.
+
+1. Go to the repo(s) you want setup.
+1. Click on the 'Settings' tab for the repo
+1. Go to the 'Webhooks' tab on this page
+1. Click the 'Add Webhook' button, and select 'Gitea'
+1. Put the endpoint URL from the previous step into the 'Target URL' box, appended with `webhook`. For example: `https://abc123.execute-api.us-east-1.amazon.aws.com/prod/webhook`
+1. Set the 'Branch filter' appropriately so that it will only select branches you want to clone when pushed to
+1. Click the 'Add Webhook' button to finish
+
+### Test the Setup
+
+If the previous steps were setup, everything should now be setup. You can now test the setup. Try pushing
+to an appropriate branch on one of the configured repos. Within the next ~30-60 seconds, the code should be
+copied to S3.
+
+You can confirm this by running the following (replacing the bucket name with the output from the deployed stack step):
+
+`aws s3 ls --recursive s3://giteatos3stack-giteacodebucket123abc-abc123/`
+
+You should see a zip file if successful.
+
+## Configure With A Codepipeline and/or Codebuild
+
+After setup is complete, you can now configure a codepipeline to pull from this code whenever it is pushed.
+
+To do this, when configuring the codepipeline source provider, select 'Amazon S3'.
+The bucket should be the name of the bucket which was output after the deploy stack step.
+The bucket key should be the full name of the repo with the appended branch name as a zip. For example:
+`organization/myProject-master.zip`
+
+The rest of the codepipeline setup should be identical to configuring any other codepipeline.
+
+Follow the same steps if you want to set up a Codebuild (which is not attached to a codepipeline)
+
+## Debugging
+
+In order to debug, you can view the logs of the lambda in Cloudwatch.
+
+If you go to the cloudwatch console and select log groups, the correct one should look something like:
+
+`/aws/lambda/GiteaToS3Stack-giteawebhookhandler123abc-abc123`
+
+You can view the logstreams in this log group to diagnose issues. Each time a webhook is received from gitea, this lambda is invoked and will log to this log group.
+
+## Useful Information
+
+This project uses [AWS CDK](https://aws.amazon.com/cdk/) to define, configure, and create the native aws application.
+
+Find more info about the cdk cli tool [here](https://docs.aws.amazon.com/cdk/v2/guide/cli.html)

UNLICENSE

@@ -0,0 +1,24 @@
+This is free and unencumbered software released into the public domain.
+
+Anyone is free to copy, modify, publish, use, compile, sell, or
+distribute this software, either in source code form or as a compiled
+binary, for any purpose, commercial or non-commercial, and by any
+means.
+
+In jurisdictions that recognize copyright laws, the author or authors
+of this software dedicate any and all copyright interest in the
+software to the public domain. We make this dedication for the benefit
+of the public at large and to the detriment of our heirs and
+successors. We intend this dedication to be an overt act of
+relinquishment in perpetuity of all present and future rights to this
+software under copyright law.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
+For more information, please refer to <http://unlicense.org>

bin/gitea-to-s3.ts

@@ -0,0 +1,9 @@
+#!/usr/bin/env node
+import 'source-map-support/register';
+import * as cdk from 'aws-cdk-lib';
+import { GiteaToS3Stack } from '../lib/gitea-to-s3-stack';
+
+const app = new cdk.App();
+new GiteaToS3Stack(app, 'GiteaToS3Stack', {
+  env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION },
+});

cdk.json

@@ -0,0 +1,31 @@
+{
+  "app": "npx ts-node --prefer-ts-exts bin/gitea-to-s3.ts",
+  "watch": {
+    "include": [
+      "**"
+    ],
+    "exclude": [
+      "README.md",
+      "cdk*.json",
+      "**/*.d.ts",
+      "**/*.js",
+      "tsconfig.json",
+      "package*.json",
+      "yarn.lock",
+      "node_modules",
+      "test"
+    ]
+  },
+  "context": {
+    "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true,
+    "@aws-cdk/core:stackRelativeExports": true,
+    "@aws-cdk/aws-rds:lowercaseDbIdentifier": true,
+    "@aws-cdk/aws-lambda:recognizeVersionProps": true,
+    "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true,
+    "@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
+    "@aws-cdk/core:target-partitions": [
+      "aws",
+      "aws-cn"
+    ]
+  }
+}

lib/gitea-to-s3-stack.ts

@@ -0,0 +1,47 @@
+import { Stack, StackProps } from 'aws-cdk-lib';
+import { Construct } from 'constructs';
+import * as s3 from 'aws-cdk-lib/aws-s3';
+import * as lambda from 'aws-cdk-lib/aws-lambda-nodejs';
+import { Runtime } from 'aws-cdk-lib/aws-lambda';
+import { Duration, CfnOutput } from 'aws-cdk-lib';
+import * as apigateway from 'aws-cdk-lib/aws-apigateway';
+
+export class GiteaToS3Stack extends Stack {
+  constructor(scope: Construct, id: string, props?: StackProps) {
+    super(scope, id, props);
+
+    const bucket = new s3.Bucket(this, 'gitea-code-bucket', {
+      versioned: true,
+    });
+    new CfnOutput(this, 'deploymentBucket', { value: bucket.bucketName });
+
+    let giteaApiUrl: string = this.node.tryGetContext('giteaApiUrl');
+    if (!giteaApiUrl) throw new Error('Context variable giteaApiUrl must be defined. cdk deploy --context giteaApiUrl=https://my.gitea/api/v1');
+    while (giteaApiUrl.endsWith('/')) giteaApiUrl = giteaApiUrl.substring(0, giteaApiUrl.length - 1);
+    if (!giteaApiUrl.endsWith('/api/v1')) throw new Error('Context variable giteaApiUrl should end with /api/v1: --context giteaApiUrl=https://my.gitea/api/v1');
+    const giteaApiToken = this.node.tryGetContext('giteaApiToken');
+    if (!giteaApiToken) throw new Error('Context variable giteaApiToken must be defined. cdk deploy --context giteaApiToken=abc123');
+    const handler = new lambda.NodejsFunction(this, 'gitea-webhook-handler', {
+      memorySize: 1024,
+      timeout: Duration.seconds(29),
+      entry: `${__dirname}/lambda-handler.ts`,
+      runtime: Runtime.NODEJS_14_X,
+      environment: {
+        GITEA_API_URL: giteaApiUrl,
+        GITEA_API_TOKEN: giteaApiToken,
+        DEPLOY_BUCKET: bucket.bucketName,
+      },
+      bundling: {
+        sourceMap: true,
+        target: 'es2018',
+      },
+    });
+    bucket.grantReadWrite(handler);
+
+    const api = new apigateway.RestApi(this, 'gitea-webhook-api', {
+      description: 'API gateway for handling gitea webhooks to clone code to S3',
+      deployOptions: { stageName: 'prod' },
+    });
+    api.root.addResource('webhook').addMethod('POST', new apigateway.LambdaIntegration(handler, { proxy: true }));
+  }
+}

lib/lambda-handler.ts

@@ -0,0 +1,67 @@
+import 'source-map-support/register';
+import * as aws from 'aws-sdk';
+import { promisify } from 'node:util';
+import { pipeline, PassThrough } from 'node:stream';
+import { promises } from 'node:fs';
+import got from 'got';
+import * as unzipper from 'unzipper';
+import * as archiver from 'archiver';
+import { APIGatewayProxyEvent, APIGatewayProxyResult, Handler } from 'aws-lambda';
+const pipelinePromise = promisify(pipeline);
+type ProxyHandler = Handler<APIGatewayProxyEvent, APIGatewayProxyResult>;
+
+interface GiteaWebhook {
+  ref: string;
+  repository: {
+    name: string;
+    full_name: string;
+  };
+}
+
+const GITEA_API_URL = process.env.GITEA_API_URL;
+const GITEA_API_TOKEN = process.env.GITEA_API_TOKEN;
+const DEPLOY_BUCKET = process.env.DEPLOY_BUCKET || '';
+
+const s3 = new aws.S3();
+
+export const handler: ProxyHandler = async (event) => {
+  console.log('Received request');
+  console.log(event.body);
+  const body: GiteaWebhook = JSON.parse(event.body || '');
+  // Only handle commits to a branch
+  if (body.ref.startsWith('refs/heads/')) {
+    const ref = body.ref.substring(11);
+    // Fetch and extract ref from webhook into temporary directory
+    const extractLocation = `/tmp/${Math.random().toString(16).substring(2, 10)}`;
+    const extract = unzipper.Extract({ path: extractLocation });
+    await pipelinePromise(
+      got.stream(`${GITEA_API_URL}/repos/${body.repository.full_name}/archive/${ref}.zip`, {
+        headers: { accept: 'application/json', Authorization: `token ${GITEA_API_TOKEN}` },
+      }),
+      extract
+    );
+    await extract.promise();
+    console.log(`Successfully downloaded and extracted to ${extractLocation}/${body.repository.name}`);
+
+    // Re-zip the contents, without a top-level folder for compatibility with codepipeline and upload to S3
+    const archive = archiver.create('zip');
+    const writeStream = new PassThrough();
+    const s3Promise = s3.upload({ Bucket: DEPLOY_BUCKET, Key: `${body.repository.full_name}-${ref}.zip`, Body: writeStream }).promise();
+    const pipe = pipelinePromise(archive, writeStream);
+    archive.directory(`${extractLocation}/${body.repository.name}/`, false);
+    await archive.finalize();
+    await s3Promise;
+    await pipe;
+    console.log(`Successfully repackaged zip and uploaded to s3://${DEPLOY_BUCKET}/${body.repository.full_name}-${ref}.zip`);
+
+    // Cleanup
+    await promises.rm(extractLocation, { recursive: true, force: true });
+    console.log('Finished cleanup');
+  } else {
+    console.log('Not a commit to a refs/head (branch). Nothing to do');
+  }
+  return {
+    statusCode: 200,
+    body: JSON.stringify({ message: 'success' }),
+  };
+};