HDDS-15945. [OEP] Support S3 WORM (Object Lock)

[chungen0126]

What changes were proposed in this pull request?

This is the OEP doc for S3 WORM (Object Lock) feature.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15945

How was this patch tested?

No tests.

[devmadhuu]

Thanks @chungen0126 for putting up the design. Curiously had a look and compared against AWS S3 specs. Got few doubts and points.

[devmadhuu]

In contrast to AWS S3, how do we present this to compliance auditors? Or should there be a "one-way" flag that, once set, cannot be cleared?

[chungen0126]

Thanks for pointing that out.

• Enabling at Creation: We can definitely support enabling Object Lock directly at bucket creation time (e.g., via the S3 CreateBucket header/parameters) to eliminate any compliance window. I updated the doc.
• One-way Flag: Exactly, objectLockEnabled is designed as a one-way flag—once enabled on a bucket, it cannot be cleared or disabled.

[devmadhuu]

Is it really true for Ozone. In AWS S3, it is being made sure to be immutable at all levels or atleast audited or blocked including admin/operators users, but in Ozone, this may not be true, currently design talks about a happy path using validateAndUpdateCache when hit on normal S3/ozone APIs. In Ozone there are other ways where someone can directly edit rocksDb or delete a key meta for a locked key. Another way, where ozone repair command also allows to restore OM's DB directory from a ratis snapshot taken before the lock was applied.

Probably the design should talk about behavior when such actions can happen in Ozone , that what is trusted boundary and upto what extent the data is protected because otherwise this design may not suffice the complete compliance audit implementation.

Can think of below:

• Whether ozone repair subcommands should refuse to operate on Raft indexes that carry lock state changes without an explicit unsafe flag.
• Whether locked keys should be detected on OM startup via cross-check between RocksDB state and a tamper-evident external audit log — so that a manual DB-copy recovery that silently removes a lock can at least be detected on the next start.
• an operator can run skip-ratis-transaction at the Raft index of a SetRetention(compliance) entry across all 3 OMs while stopped, then restart, and the audit trail of the lock is gone (though RocksDB may still hold applied state

[chungen0126]

Thanks for pointing this out!

I've added a Trusted Boundary section to clarify that any operations requiring root access bypass the application-level API and are outside the trusted boundary.

[devmadhuu]

Is it possible to move the OM leader's clock forward: The WORM check in validateAndUpdateCache will compare retainUntilDate (stored) against "now". "Now" is whatever the OM leader believes the time is.

[sumitagrawl]

Since this object Locking is from S3, what interface via s3 api will be used to configure or manage? or its directly via ozone cli?

[chungen0126]

Object locking can be configured and managed via both the Ozone API and the S3 API.

• Ozone API: It uses OzoneBucket#setRetentionConfig. If no default retention is set, the rule field within retention will simply be null.
• S3 API: It is managed through PutBucketObjectLockConfiguration, which basically follows the standard S3 specifications. You can refer to the official AWS S3 documentation for more details.

[sumitagrawl]

If we introduce this feature in Ozone for validation via above api, Having same at ranger do have any specific advantage ?

[sumitagrawl]

how ranger and ozone role will combine in validation?

[sumitagrawl]

do impact for bucket removal or volume removal or directory removal where as per soft delete, it moves to deleted directory table, but generally validation is not done to deep child ?

[chungen0126]

This can be divided into two parts:

1. Bucket or volume removal:
   Currently, a bucket cannot be deleted if it still contains keys. Therefore, this is not impacted.
2. Directory removal:
   Regarding the non-goal, the current design does not support FSO or legacy buckets, so it will not touch upon directory removal.

[sumitagrawl]

Like ranger can clear / update policy, and super admin role to update / delete bucket or file policy ?

[chungen0126]

Thanks for pointing this out!

I've added a Trusted Boundary section to clarify that any operations requiring root access bypass the application-level API and are outside the trusted boundary.

[ChenSammi]

Thanks @chungen0126 for working on this design.

It looks like bucket level retention configuration is different than object level. It support days and years, not the retainUntilDate.

https://docs.aws.amazon.com/cli/latest/reference/s3api/put-object-lock-configuration.html

[ChenSammi]

CLEAR_LEGAL_HOLD can be covered by PUT_LEGAL_HOLD, since AWS S3 doesn't have CLEAR_LEGAL_HOLD permission too, so let's avoid introduce a new extra permission.

[fmorg-git]

hi all - you won't need to create new access types for this. From the STS project, the concept of actions are created in Ranger, which map exactly to the AWS actions without the s3: prefix. From the AWS documentation (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html), there are 7 new actions:

s3:BypassGovernanceRetention
s3:GetBucketObjectLockConfiguration
s3:GetObjectLegalHold
s3:GetObjectRetention
s3:PutBucketObjectLockConfiguration
s3:PutObjectLegalHold
s3:PutObjectRetention

so in Ranger these would be:

BypassGovernanceRetention
GetBucketObjectLockConfiguration
GetObjectLegalHold
GetObjectRetention
PutBucketObjectLockConfiguration
PutObjectLegalHold
PutObjectRetention

For example, if you want the role to have access PutObjectLegalHold, you would grant PutObjectLegalHold action and write permission (i.e. access type) in Ranger. Or you could grant Put* and all permission and get access to all the Put actions.

[fmorg-git]

The mappings in Ranger for actions to permissions/access types are found here: https://github.com/apache/ranger/blob/master/security-admin/src/main/webapp/react-webapp/src/utils/actionRequirements/ozone.json.

Also, please note that when STS is merged to master, the IamSessionPolicyResolver would need to get updated with the new actions as well as S3GActionIamMapper.

[fmorg-git]

Also, like @ChenSammi says, I believe you need to separate the authorization from the value being set. For example, the same PutObjectLegalHold authorization can be used with values ON or OFF in the API (see https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectLegalHold.html)

[fmorg-git]

Also, to use the actions in Ranger, ensure to set the Ranger feature flag to true: https://ozone.apache.org/docs/next/administrator-guide/operations/s3/sts#ranger-feature-flag-action-matches-policy-condition

[ChenSammi]

There is one more S3 behavior, versioning is enabled automatically when Object Lock is enabled.

[chungen0126]

updated

[ChenSammi]

Reject upload in CreateMultipartUpload is good enough for the MPU, CompleteMultipartUpload and UploadPart can be untouched.

[chungen0126]

In a distributed development environment, it is possible for your application to initiate several updates on the same object at the same time. Your application might initiate several multipart uploads using the same object key. For each of these uploads, your application can then upload parts and send a complete upload request to Amazon S3 to create the object. When the buckets have S3 Versioning enabled, completing a multipart upload always creates a new version. When you initiate multiple multipart uploads that use the same object key in a versioning-enabled bucket, the current version of the object is determined by which upload started most recently (createdDate).

AWS S3 explicitly allows multiple concurrent multipart uploads for the same key.

If we only reject requests at CreateMultipartUpload, an MPU initiated before the lock is applied could still be completed after the lock becomes active. This would overwrite the protected object and break the WORM guarantee. Therefore, we still need to validate the WORM state during CompleteMultipartUpload to ensure strict immutability.

[ChenSammi]

I think the major reason is we have added new OM APIs for the feature, so we need a new OM version to tell client whether these APIs can be called or not. And table wise, since all new fields are optional fields, they are naturally backward compatible.

[chungen0126]

updated

[ChenSammi]

Do we have an JIRA open in Ranger project to track all the changes required for this feature?

[ChenSammi]

@fmorg-git , would you mind sharing some knowledge with @chungen0126 about how to expose new permissions for Ranger, and what are the major tasks needed for Ranger integration, including UI changes?

[fmorg-git]

I added comments

[ChenSammi]

Why BYPASS_GOVERNANCE is configured at volume level?

[fmorg-git]

I believe BypassGovernanceRetention should be checked at the key level.

[chungen0126]

Thanks @ChenSammi, @devmadhuu @sumitagrawl, and @fmorg-git for the review! I’ve updated the design doc based on your feedback—please take another look.

Regarding the Ranger integration part, I'm still relatively new to it and actively catching up, so I would really appreciate it if @fmorg-git could help take a look. I'll continue refining the Ranger-related details in upcoming updates.

[devmadhuu]

@chungen0126 , just want to highlight to be in compliance with AWS S3 WORM, that AWS has recently added variable retention with event holds to S3 Object Lock, so may be we can think about adding EVENT_TRIGGERED_RETENTION in our existing enum for extended support. Enterprise S3 semantics are increasingly about governance workflows, not just object durability. Reference : AWS S3 doc

object created
      |
event hold
      |
business event occurs
      |
hold released
      |
retention timer starts
      |
WORM until timer expires

[chungen0126]

@chungen0126 , just want to highlight to be in compliance with AWS S3 WORM, that AWS has recently added variable retention with event holds to S3 Object Lock, so may be we can think about adding EVENT_TRIGGERED_RETENTION in our existing enum for extended support. Enterprise S3 semantics are increasingly about governance workflows, not just object durability. Reference : AWS S3 doc

object created
      |
event hold
      |
business event occurs
      |
hold released
      |
retention timer starts
      |
WORM until timer expires

Thanks @devmadhuu for the reminder. I add a sub-section in Terminology to introduce the trigger workflow.

[chungen0126]

@devmadhuu @ChenSammi @sumitagrawl PTAL

[fmorg-git]

nit - I believe Put requires CREATE and WRITE access.

[fmorg-git]

quick clarification - does it mean BypassGovernanceRetention requires WRITE and DELETE together, similar to how PutObject requires WRITE and CREATE together? If so, may want to say and instead of /

[fmorg-git]

curious, what is the reason for this additional complexity? AWS doesn't work like this. FYI @ChenSammi

[chungen0126]

Thanks for the feedback! Actually, AWS IAM does support this kind of value-level condition matching for S3.

According to the AWS IAM Condition Operators documentation, S3 provides condition keys that allow restricting specific actions based on request values.

The reason I included this additional complexity is based on use cases like the one described in this AWS re:Post article: Denying removal of Amazon S3 Object Lock Legal Hold. The article demonstrates how administrators separate the permissions for turning a legal hold ON versus OFF by leveraging the StringEquals condition operator. As mentioned in the quote:

As legal hold has no expiration date, users may wish to use this mode to apply an indefinite lock on objects they wish to protect from accidental or malicious deletion. In this scenario, it may be desirable to restrict permissions to remove legal hold from objects. You can do this with a condition key in the Condition element of an IAM policy, specifically "StringEquals": "s3:object-lock-legal-hold": "OFF" applied to the action "PutObjectLegalHold".

That being said, I am completely open to discussing this. If we feel that PutObjectLegalHold (without distinguishing between ON/OFF payloads) is enough for Ozone at this stage, we can definitely drop this custom condition evaluator to keep the initial implementation simple.

Let me know what @fmorg-git and @ChenSammi think!

[fmorg-git]

thanks for the references @chungen0126 - wasn't aware of it. I implemented Object Lock in the past and hundreds of thousands of customers never needed/asked for it, so I was not aware (also I noticed the AWS article was posted after the implementation). My gut feeling is the complexity required might be overkill compared to how many people might use it (if any). I also note that we currently don't support Deny in STS session policy nor any condition key other than s3:prefix. In any case, I will defer to @ChenSammi to for her thoughts if it is needed for Ozone.

[shuan1026]

These two classes don't exist in master. The path is BucketEndpoint.multiDelete -> OzoneBucket.deleteKeys -> OMKeysDeleteRequest. Checking in S3G also misses KeyLifecycleService, which submits DeleteKeysRequest directly into Ratis and never goes through S3G. Could the check go in OMKeysDeleteRequest.validateAndUpdateCache?

[jojochuang]

Summary

Doc-only OEP for S3 Object Lock on OBS buckets (Ranger + STS action matcher, single-version v1). CI looks good. Approve — scope, two-phase OM enforcement (preExecute + validateAndUpdateCache), and the trusted-boundary section are strong and actionable.

OEP vs AWS S3 Object Lock (v1)

• Bucket scope: OBS only (no FSO/legacy). AWS applies to versioned S3 buckets generally.
• Versioning: AWS requires versioning; locks are per object version. This OEP v1 is single-version — locks apply to the current key; multi-version lock is Phase 6.
• Overwrites: AWS typically creates a new version; the locked version stays. OEP rejects Put/Copy/MPU on a locked key (stricter without versioning).
• Deletes: AWS is version-aware (403 on protected version; delete markers in some paths). OEP blocks delete on protected keys in v1 (no delete-marker model).
• Enable lock on bucket: Both support create-time and existing-bucket config; AWS uses x-amz-bucket-object-lock-token on PutObjectLockConfiguration — please document that token flow explicitly in the OEP (enable-on-existing-bucket is AWS-aligned).
• Modes / bypass: Governance + Compliance + Legal Hold align with AWS; BypassGovernanceRetention + x-amz-bypass-governance-retention: true; no bypass for Compliance or Legal Hold on delete.
• Auth: Lock admin APIs assume Ranger + enableActionMatcherInPoliciesCondition, not native Ozone ACLs.
• Compliance story: Infra bypass (RocksDB, repair, DN files, clock, Ranger admin) is appropriate for self-managed Ozone vs AWS-managed S3.

Main client-visible deltas in v1: no versioning, overwrite blocking, and Ranger dependency — worth a short callout in Summary or Compatibility for operators/SDK authors.

Follow-ups (track on HDDS-15945)

1. Add CreateBucket Object Lock and PutObject / MPU x-amz-object-lock-* headers to the “Supported S3 APIs” section (Phase 4 already covers behavior).
2. Align Key table narrative with RetentionConfig / effective RetainUntilDate (Table Changes still mentions retentionDate).
3. Use optional proto fields consistently with the Compatibility section (required bool objectLockEnabled in Phase 1 is contradictory for rolling upgrade).
4. Document Ranger repo coordination (ozone.json, condition evaluators) and non-Ranger clusters (no lock admin path in v1).
5. Fix plan wording (five phases vs Phase 6); minor Java sketch (boolean not bool).

Feature checklist (requested product scope)

Per-bucket Object Lock, per-object retention, Legal Hold, compliance enforcement, and “admin cannot delete” (Compliance / legal hold via API path) are covered by this design. AWS API parity is mostly there for the six lock REST actions + bypass; gaps are mainly CreateBucket, versionId, and token for existing buckets.

submitted wrong

[jojochuang]

Future: FSO buckets and Object Lock

We may need to support FILE_SYSTEM_OPTIMIZED (FSO) buckets eventually. The current OEP correctly scopes v1 to OBS, but it would help to add a short “Future: FSO” note so readers do not assume OBS handlers generalize for free.

FSO is not natively compatible with the same WORM story as flat OBS keys. AWS Object Lock is S3-object/version oriented; FSO adds a file/directory split, path-based identity, rename, recursive/batch delete, and background purge (DirectoryDeletingService, OMDirectoriesPurgeRequestWithFSO, etc.). Several of those paths have no AWS S3 analogue, so product and compliance rules need explicit decisions before implementation.

Design decisions to make

• Scope of lock: Files only (FileTable / OmKeyInfo) vs directory entries (DirectoryTable / OmDirectoryInfo).
• Clients: S3G only vs also OFS/Ozone FS (rename/delete/overwrite bypass S3 action mapping unless OM enforces WORM on every RPC).
• Rename / move: Block when locked vs allow and carry lock metadata on the same objectID; rename-over-existing-key vs Put overwrite rules.
• Tree delete: Prefix delete when a descendant is locked; align partial failure with DeleteObjects semantics.
• Legacy FSO buckets vs current FSO layout before allowing objectLockEnabled.

Engineering pointers (when FSO is in scope)

1. Duplicate handler surface: WORM must be wired into *WithFSO create/commit/delete/MPU paths and OMFileCreateRequestWithFSO, not only OBS OMKey*Request. A shared WormPolicyEvaluator (or similar) reduces risk of one missed handler becoming a bypass.
2. Background jobs are high risk: Directory/key purge and pending-deletion services must respect lock state; async purge must not delete locked files under “deleted” directory markers.
3. Rename: OMKeyRenameRequestWithFSO / batch rename — treat as first-class WORM surface (DELETE+CREATE ACL-wise; compliance impact if rename is allowed without policy).
4. Enforcement authority: OM validateAndUpdateCache (and equivalent on FSO handlers) should remain authoritative; Ranger governs lock administration and governance bypass, not replacement for metadata checks on OFS clients.
5. Testing: FSO-specific integration — tree delete with one locked file, rename, MPU on paths, purge service, optional o3fs if in scope.

Suggest a brief Non-goal today / Future work subsection in the OEP pointing here so OBS v1 can ship without implying FSO is a small delta.

[jojochuang]

Observability / displaying WORM state

The OEP covers querying lock configuration through the standard S3 Object Lock Get APIs (GetBucketObjectLockConfiguration, GetObjectRetention, GetObjectLegalHold) and Phase 3 OM / metadata-reader paths. That matches how AWS expects clients to read bucket default config and per-object retention / legal hold when they call those sub-resources explicitly (AWS CLI, Boto3, etc.).

For seeing lock state on normal object reads, AWS documents HeadObject and GetObject response headers (when the caller has s3:GetObjectRetention / s3:GetObjectLegalHold):

• x-amz-object-lock-mode
• x-amz-object-lock-retain-until-date
• x-amz-object-lock-legal-hold
• (and event-hold related headers where variable retention applies)

The OEP does not yet call out enriching Head/Get with those headers. That is the main S3 parity gap for “display WORM state without extra GET ?retention / ?legal-hold round trips.”

Operator tooling: The doc does not discuss ozone sh / admin / Recon (or similar) showing objectLockEnabled, default retention, or effective protection on a key. The Java ObjectStore snippet shows setters only; read paths outside the six S3 Get handlers are not spelled out.

Suggest a short “Observability” subsection:

• v1 minimum (AWS-aligned): the three Get APIs + document HeadObject/GetObject lock headers and required Ranger Get* actions.
• Follow-ups (product): CLI/Recon and read APIs on ObjectStore if non-S3 clients need lock metadata.