GitHub - joevizcara/pulumi-aws
github.com
Contribute to joevizcara/pulumi-aws development by creating an account on GitHub.

i am not a devops engineer. i appreciate any critique or correction.

code: gitlab github

Deploying Nextcloud on AWS ECS with Pulumi

This Pulumi programme deploys a highly-available, cost-effective Nextcloud service on AWS Fargate with a serverless Aurora PostgreSQL database.

Deployment Option 1 (GitOps)

The first few items are high-level instructions only. You can follow the instructions from the hyperlinked web pages. They include the best practices as recommended by the authors.

  1. A Pulumi account. This is for creating a Personal Access Token that is required when provisioning the AWS resources.
  2. Create a non-root AWS IAM User called pulumi-user.
  3. Create an IAM User Group called pulumi-group
  4. Add the pulumi-user to the pulumi-group User Group.
  5. Attach the IAMFullAccess policy to pulumi-group. The IAMFullAccess allows your IAM User to add the remaining required IAM policies to the IAM User Group using the automation script later.
  6. Create an access key for your non-root IAM User.
  7. On your Pulumi account, go to Personal access tokens and create a token.
  8. Also create a password for the Aurora Database. You can use a password generator.
  9. Clone this repository either to your GitLab or GitHub.
  10. This works either on GitLab CI/CD or GitHub Actions. On GitLab, go to the cloned repository settings > Settings > Variables. On GitHub, go to the cloned repository settings > Secrets and variables > Actions > Secrets.
  11. Store the credentials from steps 6-8 as AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, PULUMI_ACCESS_TOKEN, and POSTGRES_PASSWORD. These will be used as environment variables by the deployment script.
  12. On AWS Console, go to EC2 > Load Balancers. The DNS name is where you access the Nextcloud Web Interface to establish your administrative credentials.

[!NOTE] The automatic deployment will be triggered if there are changes made on the main.go, .gitlab-ci.yml, or the ci.yml file upon doing a git push. On main.go, you can adjust the specifications of the resources to be manifested. Notable ones are in lines 327, 328, 571, 572, 602, 603, 640.

Deployment Option 2 (Manual)

  1. Install Go, AWS CLI, and Pulumi.
  2. Follow steps 1-8 above.
  3. Add the required IAM policies to the IAM User Group to allow Pulumi to interact with AWS resources:
printf '%s\n' "arn:aws:iam::aws:policy/AmazonS3FullAccess" "arn:aws:iam::aws:policy/AmazonECS_FullAccess" "arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess" "arn:aws:iam::aws:policy/CloudWatchEventsFullAccess" "arn:aws:iam::aws:policy/AmazonEC2FullAccess" "arn:aws:iam::aws:policy/AmazonVPCFullAccess" "arn:aws:iam::aws:policy/SecretsManagerReadWrite" "arn:aws:iam::aws:policy/AmazonElasticFileSystemFullAccess" "arn:aws:iam::aws:policy/AmazonRDSFullAccess" | xargs -I {} aws iam attach-group-policy --group-name pulumi-group --policy-arn {}
  1. Add the environment variables.
export PULUMI_ACCESS_TOKEN="value" && export AWS_ACCESS_KEY_ID="value" && export AWS_SECRET_ACCESS_KEY="value" && export POSTGRES_PASSWORD="value"
  1. Clone the repository locally and deploy.
mkdir pulumi-aws && \
cd pulumi-aws && \
pulumi new aws-go && \
rm * && \
git clone https://gitlab.com/joevizcara/pulumi-aws.git . && \
pulumi up

Deprovisioning

pulumi destroy --yes

Local Testing

The Pulumi.aws-go-dev.yaml file contains a code block to use with Localstack for local testing.

Features

  1. Subscription-free application - Nextcloud is a free and open-source cloud storage and file-sharing platform.
  2. Serverless management - using Fargate and Aurora Serverless reduces infrastructure management.
  3. Reduced cost - can be scaled and as highly available as an AWS EKS cluster, but with cost lower per-hour.
  4. Go coding language - a popular language for cloud-native applications, eliminating syntax barriers for engineers.

Diagramme

  • magikmw@piefed.socialEnglish
    2·
    13 hours ago

    All comments about overkill are amusing. You do you. Did you learn stuff?

    Maybe you can replace some of those tools with less expensive analogs, how’s the cost anyway?

      • Aurelian@lemmy.mlEnglish
        1·
        7 hours ago

        If you want it to stand out don’t automate the compute and networking that’s so standardized these days that anyone can do it, Automated those IAM permissions.

        I know that when hiring nothing gets me more excited about a candidate than them understanding how to securely bootstrap an environment.

      • loudwhisper@infosec.pubEnglish
        1·
        9 hours ago

        In that case, Pulumi permissions are too broad IMHO for what it has to do, an enterprise should adhere to least privilege. Likewise, as I wrote in another comment, the egress security groups are unclear to me (why any traffic at all is needed?) and the image consumed should be pinned to a digest. Or better yet, should be coming from a private enterprise registry, ideally with an attestation that can be verified at runtime.

        I am not sure ECS Fargate makes sense vs an ec2 instance to run the workload. This setup alone will cost about $30/month assuming half a vCPU per replica with Fargate, plus about $12 for the memory (1GB/task). 2xt2.micro could be run for ~$20 without even considering reservation discounts etc. Obviously the gap will become even larger at scale, which I suppose might be very interesting for an enterprise.

      • cichy1173@szmer.infoEnglish
        21·
        12 hours ago

        If that’s true, then great, and learning with cloud-native technology is perfectly fine. The critical comments were probably made because the post does not indicate that this is just an idea for production architecture or a form of learning, but rather the actual deployment that should be carried out (at least that is how I see it), which in this subreddit could be perceived as a proposal for self-hosting for private individuals (as self-hosting is associated with private individuals).

    • cichy1173@szmer.infoEnglish
      31·
      13 hours ago

      I started counting, and only with Fargate two ECS tasks (with not much CPU power) and with first Aurora DB it is almost 200 USD per month (in Frankfurt). If we will add another services, the cost will be higher and higher.

      Did you learn stuff?

      Yeah, learning is great and if you will deploy it and kill it in the same day, the cost will be quite low. But if you want to really use it, it is too much, it is better to use hostable alternatives, ie Load Balancer == Haproxy, Fargate Task == Docker on EC2/VPS (even with ECS), Aurora == burstable tier RDS or DB hosted on VPS/EC2. I know, in business area, you should not host DB on EC2 or use clean Docker on EC2 (without ECS) (and that production Nextcloud deployment could be more extended, because availability and scalability is more important that saving some dollars), but in private zone, where every penny is important, it is overkill for everyday use.

  • loudwhisper@infosec.pubEnglish
    18·
    22 hours ago

    Everyone is free to pick their poison, but I have to ask…why? What is the target audience here? This is a massively overkill architecture IMHO. Not to talk about the fact you now need 3 managed services (fargate, s3 and aurora at least) for a single self hosted tool, and that is being generous (not counting cloudwatch, ALBs, etc.).

    • Why do you need security groups to allow egress anywhere (or, at all)?
    • I would pin the image to a digest, rather than using latest.
    • what is the average monthly cost for this infra for you?
    • cichy1173@szmer.infoEnglish
      3·
      18 hours ago

      Not that cheap. Both Aurora and Fargate can be pricy, so using this for personal cloud, not as business solution, is not only a overkill, but also expensive tool, that you will not fully reuse for other services. I think, in personal selfhosted area, we agree to not use that overkilled architecture (but for typical cloud deployment it is fairly simple) to cut costs massively.

      • loudwhisper@infosec.pubEnglish
        2·
        15 hours ago

        Oh yeah, I am aware. Mostly here I would question the idea to have multi-AZ redundancy and using a manage service for DB (which indeed is expensive). All of this when a 5$ VPS could host the same (maybe still using s3 for storage) and accept the few hours downtime in the rare event your VPS explodes and you need to restore it from a backup.

        So from my PoV this is absolutely overkill but I concede that it depends a lot on the requirements. I can’t ever imagine having requirements so tight that need such infra to run (in fact, I think not even most businesses have these requirements, I have written on the topic at https://loudwhisper.me/blog/hating-clouds/) for my personal stuff…

        • cichy1173@szmer.infoEnglish
          2·
          14 hours ago

          Yes, just like I said, when running it for personal use, going with SLA 99,(9) is too expensive. As far as long we say about serverless solutions, they can be great and helpful (I can say that from both SysOps and DevOps perspective that work on many projects), but I don’t think they should be used in homelab form, as they do not provide that much customisations, and homelabs are the place where we want to experiment and have some fun, not just deploy something in a way that will “just work”.

          • loudwhisper@infosec.pubEnglish
            1·
            13 hours ago

            Plus, at this point why not using directly managed Nextcloud (or alternatives)… If anyway you use a managed storage, runtime and database, in a vendor lock…

            • cichy1173@szmer.infoEnglish
              1·
              13 hours ago

              I would not go with managed NC, because you can’t control nothing and provider raise prices over and over. Even with serverless Nextcloud deployment, the architecture is still like LEGO, and if something will go bad or price will be too high, then you can exchange those LEGO bricks, ie. migrate from Fargate to EC2 w/ ECS, migrate from Aurora to RDS Postgres or Postgres installed on EC2 and so one.

  • danAEnglish
    4·
    19 hours ago

    This seems like overkill compared to just running it on a VPS and having a second VPS as a hot spare.

  • Rimu@piefed.socialEnglish
    6·
    22 hours ago

    One availability zone is enough. I am not convinced that Aurora is good value.

    This seems like an ad for pulumi, whatever that is.