Skip to content

Latest commit

 

History

History
118 lines (91 loc) · 5.33 KB

README.md

File metadata and controls

118 lines (91 loc) · 5.33 KB

Amazon S3

S3 Glacier Vault Lock

  • Adopt a WORM (Write Once Read Many) model
  • Create a Vault Lock Policy
  • Lock the policy for future edits (can no longer be changed or deleted)
  • Helpful for compliance and data retention

Glacier Vault

S3 Object Lock (versioning must be enabled)

  • Adopt a WORM (Write Once Read Many) model
  • Block an object version deletion for a specified amount of time
  • Retention mode - Compliance:
    • Object versions can't be overwritten or deleted by any user, including the root user
    • Objects retention modes can't be changed, and retention periods can't be shortened
  • Retention mode - Governance:
    • Most users can't overwrite or delete an object version or alter its lock settings
    • Some users have special permissions to change the retention or delete the object
  • Retention Period: protect the object for a fixed period, it can be extended
  • Legal Hold:
    • protect the object indefinitely, independent from retention period
    • can be freely placed and removed using the s3:PutObjectLegalHold IAM permission

Vault Policies & Vault Lock

  • Each Vault has:
    • ONE vault access policy
    • ONE vault lock policy
  • Vault Policies are written in JSON
  • Vault Access Policy is like a bucket policy (restrict user / account permissions)
  • Vault Lock Policy is a policy you lock, for regulatory and compliance requirements.
    • The policy is immutable, it can never be changed (that’s why it’s call LOCK)
    • Example 1: forbid deleting an archive if less than 1 year old
    • Example 2: implement WORM policy (write once read many)

Glacier Vault Policies

Vault Lock Process

Glacier – Vault Lock Process

Moving between Storage Classes

  • You can transition objects between storage classes
  • For infrequently accessed object, move them to Standard IA
  • For archive objects that you don’t need fast access to, move them to Glacier or Glacier Deep Archive
  • Moving objects can be automated using a Lifecycle Rules

Moving Storage Classes

Lifecycle Rules

  • Transition Actions – configure objects to transition to another storage class
    • Move objects to Standard IA class 60 days after creation
    • Move to Glacier for archiving after 6 months
  • Expiration actions – configure objects to expire (delete) after some time
    • Access log files can be set to delete after a 365 days
    • Can be used to delete old versions of files (if versioning is enabled)
    • Can be used to delete incomplete Multi-Part uploads
  • Rules can be created for a certain prefix (example: s3://mybucket/mp3/*)
  • Rules can be created for certain objects Tags (example: Department: Finance)

Lifecycle Rules (Scenario 1)

  • Your application on EC2 creates images thumbnails after profile photos are uploaded to Amazon S3. These thumbnails can be easily recreated, and only need to be kept for 60 days. The source images should be able to be immediately retrieved for these 60 days, and afterwards, the user can wait up to 6 hours. How would you design this?
  • S3 source images can be on Standard, with a lifecycle configuration to transition them to Glacier after 60 days
  • S3 thumbnails can be on One-Zone IA, with a lifecycle configuration to expire them (delete them) after 60 days

Lifecycle Rules (Scenario 2)

  • A rule in your company states that you should be able to recover your deleted S3 objects immediately for 30 days, although this may happen rarely. After this time, and for up to 365 days, deleted objects should be recoverable within 48 hours.
  • Enable S3 Versioning in order to have object versions, so that “deleted objects” are in fact hidden by a “delete marker” and can be recovered
  • Transition the “noncurrent versions” of the object to Standard IA
  • Transition afterwards the “noncurrent versions” to Glacier Deep Archive

Amazon S3 Analytics

Storage Class Analysis

  • Help you decide when to transition objects to the right storage class
  • Recommendations for Standard and Standard IA
  • Does NOT work for One-Zone IA or Glacier
  • Report is updated daily
  • 24 to 48 hours to start seeing data analysis
  • Good first step to put together Lifecycle Rules (or improve them)!

Storage Class Analysis

Replication (CRR & SRR)

  • Must enable Versioning in source and destination buckets
  • Cross-Region Replication (CRR)
  • Same-Region Replication (SRR)
  • Buckets can be in different AWS accounts
  • Copying is asynchronous
  • Must give proper IAM permissions to S3
  • Use cases:
    • CRR – compliance, lower latency access, replication across accounts
    • SRR – log aggregation, live replication between production and test accounts

Replication

  • After you enable Replication, only new objects are replicated
  • Optionally, you can replicate existing objects using S3 Batch Replication
    • Replicates existing objects and objects that failed replication
  • For DELETE operations
    • Can replicate delete markers from source to target (optional setting)
    • Deletions with a version ID are not replicated (to avoid malicious deletes)
  • There is no “chaining” of replication
    • If bucket 1 has replication into bucket 2, which has replication into bucket 3
    • Then objects created in bucket 1 are not replicated to bucket 3