Skip to content

CALCLOUD-462 Increase disk size for reprocessing MVMs - #220

Closed
sashawood wants to merge 9 commits into
mainfrom
release/v0.4.49
Closed

CALCLOUD-462 Increase disk size for reprocessing MVMs#220
sashawood wants to merge 9 commits into
mainfrom
release/v0.4.49

Conversation

@sashawood

Copy link
Copy Markdown
Collaborator

Same as #219 plus:

  1. black formatting
  2. Better commenting of the connection between LENGTH_LADDER in deploy.sh and length(ladder) in locals.tf.

Previously, the terraform scripts:

created 1 launch template called hstsdp with a disk size of 150GB.
used a variable called local.ladder with 4 rungs - 02g, 08g, 16g, 64g
created 4 job queues, compute environments, and job definitions - one each for each of the 4 local.ladder rungs

I have modified the scripts to:

create 2 launch templates: hstdp["default"] with a disk size of 150GB and hstdp["large-volume"] with a disk size of 300GB.
use a local.ladder var with 8 rungs - 02g, 08g, 16g, 64g and 02g-large-volume, 08g-large-volume, 16g-large-volume, 64g-large-volume
create 8 job queues, compute environments, and job definitions - one each for each of the local.ladder rungs

In addition,

I modified multiple places where we explicitly used to iterate over 4 rungs of local.ladder to now use iterate over length(local.ladder) rungs.
In plan.py, I modified functions to use "default" AWS resources for ipppssoots and svms, and "large-volume" resources for mvms.
I updated the tests to include the new queues, compute environments, and definitions.

I have run this with tests 08106 and 08107 and observed the ipppssoots and svms are routed to default volume size EC2 instances, and mvms are routed to large-volume EC2 instances.
https://jira.stsci.edu/browse/CALCLOUD-462?focusedId=1037304&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1037304

sashawood and others added 9 commits June 24, 2026 12:26
Do not pass the output of `aws ec2 describe-images` on the command line.
It is too long for the OS to handle.  Instead, pass it through a file.
CALCLOUD-464 Make amirotate codebuild script work again
* In aws_s3_bucket move server_side_encryption configuration to aws_s3_bucket_server_side_encryption_configuration
* When using awsudo $ADMIN_ARM aws, do not omit the aws command.
* Taint aws_batch_compute_environments in a loop.
* Install repos if /tmp/calcloud_install dirs do not exist.
@sashawood
sashawood requested a review from raswaters July 17, 2026 20:09
@sashawood

Copy link
Copy Markdown
Collaborator Author

Closing without merge, because this was created due to confusion.

@sashawood sashawood closed this Jul 17, 2026
@sashawood
sashawood deleted the release/v0.4.49 branch July 17, 2026 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant