CIP-0168? | More BuiltinValue Functions

[fallen-icarus]

CIP-0153 added a new BuiltinValue type to make working with on-chain values more efficient. However, it added only a few builtin functions for working with this new BuiltinValue type which limits its real world usability. Adding and maintaining builtin functions is costly, but the importance of validating values in the eUTxO model justifies a wider range of builtin functions for this purpose. With this in mind, this CIP proposes a few extra builtin functions to improve the usability of this new BuiltinValue type while still trying to minimize the overall maintenance burden.

---

(Rendered Version)

[fallen-icarus]

Tagging people who might be interested in weighing in:

@lehins @WhatisRT @zliu41 @effectfully @MicroProofs @colll78 @ana-pantilie @Unisay @Quantumplation @rvcas

[fallen-icarus]

@effectfully With respect to your reply here about intersectionValue and flattenValue:

I agree with you on flattenValue just being valueData. I usually only use aiken's flatten for pattern matching and valueData covers that use case.

However, for using intersection for lookupTokens, I use aiken's tokens a lot in my CIP-89 protocols so the extra overhead would quickly add up. I'm not a software engineer, but IMO lookupTokens is just a simple key lookup so I don't see there being much of a burden from supporting it as a separate builtin. How is it much different than lookupCoin?

[zliu41]

• We'll need mkEmptyValue as a primitive, otherwise in Plutus V4 there will be no way to construct an empty value.
• What's the motivation for negateValue?
• It seems useful to generalize policies to return [(CurrencySymbol, [(TokenName, Integer)])]?
• What's the motivation for intersection? I think it can be done without a builtin, assuming we add all other builtins proposed, so this will need strong justification.

[fallen-icarus]

We'll need mkEmptyValue as a primitive ...

I had assumed this was covered by unValueData $ List []. What would valueData return if used on an empty BuiltinValue?

What's the motivation for negateValue?

Subtracting values. unionValue only adds two values. I can use negateValue on one and then unionValue will do subtraction. This is useful for calculating the change to return at runtime.

... generalize policies to return [(CurrencySymbol, [(TokenName, Integer)])]?

policies is meant to only return the policy ids in the BuiltinValue like aiken's version. It is effectively just the keys function from Data.Map in Haskell. What you are suggesting just seems like valueData which would be more costly since it is processing extra unneeded information.

What's the motivation for intersection?

I may have gotten ahead of myself on this one. The idea was to have a single builtin support arbitrary higher-level filtering functions like aiken's restricted_to. But I'm not sure now if it will actually work the way I was thinking...

[zliu41]

I had assumed this was covered by unValueData $ List []. What would valueData return if used on an empty BuiltinValue?

It's actually unValueData $ Map []. But that's only the case for Plutus V1 through V3. From V4, there will be a new Value constructor for Data, so Value will no longer be encoded as a Map. That said, I don't think emptyValue is actually necessary since we can just use a constant, so I take it back.

What you are suggesting just seems like valueData

This is also only the case for Plutus V1 through V3. From V4, valueData will essentially be a no-op: just wrapping a Value within Data's Value constructor.

The broader problem with Value is this: there are a lot of useful operations on nested maps, any subset of them can be made primitives, and the possibilities are almost endless. So I'd want to proceed very carefully, and give people enough time to think it through and reach consensus that it is the right primitive.

For example, if we have a valueToList primitive that returns [(CurrencySymbol, [(TokenName, Integer)])], then many other primitives can be done without primitives. So to make a case that a particular primitive should be a primitive, it will be very helpful to, for example, lay out some concrete use cases where the primitive leads to significant performance improvements.

[rphair]

Accepted as CIP candidate at today's CIP meeting based mainly on initial Plutus stakeholder response being queries about the specifics, rather than any claim of infeasibility or disinterest... and because support for this idea was also shown here although not included with the recent CIP-0153 update:

• #1088

@fallen-icarus please update the link to the current rendered document & change the directory name to CIP-0168 🎉

[michaelpj]

This seems pretty reasonable to me. It would be nice to see the motivation expanded with examples.

[michaelpj]

To be a bit annoying: it would make this proposal much more concrete to have an appendix listing all the interesting functions you want to implement with their implementations using the new primitives. I think they're presumably all quite simple, and it would make the case quite compelling.

Ideally we'd also be able to see how this improves the costing behaviour of these functions.

[michaelpj]

I find the naming a bit confusing since it returns the same type it is given. How about restrictPolicyTo?

[fallen-icarus]

Perhaps it is better to make restrictPolicyTo be:

restrictPolicyTo :: [BuiltinCurrencySymbol] -> BuiltinValue -> BuiltinValue

This would support a higher-level lookupTokens as well as aiken's restricted_to (here).

[michaelpj]

Adding negate to union gives us a group on Value. The obvious missing thing to me is scalar multiplication (to give us a module). Does anyone need that?

[fallen-icarus]

I don't personally have a use for scalar multiplication (right now), but I could see it being useful to others.

[effectfully]

Is negate just scalar multiplication with the scalar being -1? In that case I think instead of adding negate we should just add scale?

[rphair]

@colll78 could this do what you need to do for #1090 (comment)?

[colll78]

It could, but it would be significantly less efficient than negateValue (roughly 64% more expensive) if we are to assume there is a similar costing difference to multiplyInteger -1 n and subtractInteger 0 n

[effectfully]

These two builtins only perform the respective arithmetic operations, scaleValue will have to traverse the given Value, which is going to make the discrepancy less pronounced.

There's a different point however: multiplyInteger was surprisingly tricky to cost, so that's gonna propagate into scaleValue (so much so that it may not even be feasible to implement that with the current costing machinery, not sure), while negateValue would be very straightforward. @kwxm what do you think?

If Kenneth agrees costing scaleValue is hard, then I personally don't mind adding negateValue right away, given your reasoning here, because even with faster caseList and casePair, the derivative negateValue is probably going to be at least an order of magnitude more expensive than a builtin one (not least because negateValue is guaranteed to produce a well-formed Value so we don't need to check any of the invariants (except for quantities being in their range), while a derivative implementation will have to, because we don't provide an unsafe builtin that could create a Value while avoiding the checks).

[zliu41]

multiplyInteger -1 n is 64% more expensive than subtractInteger 0 n? If that's the case I'd be surprised, and it would suggest that either multiplyInteger or subtractInteger is probably not costed properly.

Regarding costing of multiplyInteger in general - since we cap the range of quantities, we should be able to consider it constant time.

Whether it's negateValue or scaleValue, it will be linear whether it's a builtin or not. Certainly the constant factors will differ greatly, but a builtin would be much more useful if it is asymptotically more expensive without it. If we are going to add one more builtin for the intra-era HF, are we absolutely sure that it should be negateValue / scaleValue?

[michaelpj]

What about getting the token names?

[fallen-icarus]

That doesn't seem as useful to me since assets could have the same token name despite having different policy ids.

[colll78]

I believe I made a huge mistake not including negateValue in the original CIP. negateValue is required for a huge number of use-cases of unionValue (to subtract one value from another). Can anyone evaluate the scope of this builtin? Is it possible we could introduce it? It is undoubtably the single most critical value builtin I neglected.

Without negateValue, the implementation of the programmable tokens CIP will not be possible, so this is very high priority for me.

Even if we need time to evaluate the demand and consider alternatives for some of the value operations proposed in this CIP, I believe negateValue is a special case where the need for it is very clear, and we have very high confidence that this is indeed a primitive value operation that we will need and it is very general.

[zliu41]

@colll78 in Plutus V1 through V3 the builtin Value will be encoded as a Map in Data, so negateValue (and indeed anything else) can be done without a builtin. So even if it doesn't make it into the intra-era HF, this shouldn't be a deal breaker for most applications, should it? We can certainly add something like negateValue for V4.

As you can see, there's a suggestion above regarding generalizing negateValue into scaleValue. So it may take some time to decide what the best option is, even though the implementation of negateValue itself may not be difficult. It's also not clear whether there are more options.

[Unisay]

Did you mean

Suggested change

	policies :: BuiltinValue -> List [BuiltinCurrencySymbol]
	policies :: BuiltinValue -> List BuiltinCurrencySymbol

or

Suggested change

	policies :: BuiltinValue -> List [BuiltinCurrencySymbol]
	policies :: BuiltinValue -> [BuiltinCurrencySymbol]

?

[fallen-icarus]

Yes, but I'm not sure what is the right syntax for Data . I went off of CIP-0153.

[colll78]

It should be BuiltinList BuiltinCurrencySymbol where type BuiltinCurrencySymbol = BuiltinData

[colll78]

@colll78 in Plutus V1 through V3 the builtin Value will be encoded as a Map in Data, so negateValue (and indeed anything else) can be done without a builtin. So even if it doesn't make it into the intra-era HF, this shouldn't be a deal breaker for most applications, should it? We can certainly add something like negateValue for V4.

As you can see, there's a suggestion above regarding generalizing negateValue into scaleValue. So it may take some time to decide what the best option is, even though the implementation of negateValue itself may not be difficult. It's also not clear whether there are more options.

negateValue can indeed be done without a builtin, but it is prohibitively expensive. In collaboration with IOG I have been working to introduce programmable assets to Cardano (https://github.com/input-output-hk/wsc-poc), the framework itself is complete, all features are implemented, and it is fully functional; the only reason that the feature is not live in production for end users is that the contracts are prohibitively expensive for mainnet production use. I have done an extensive amount of benchmarking, and the bottleneck is the value operations, particularly unionValue and negateValue, these are the two most expensive operations currently. The issue with the non-builtin implementation of negateValue is that it must traverse the entire value structure:

-- | Negate the quantity of every token in a Value
negateValue :: BuiltinData -> BuiltinData 
negateValue = BI.mkMap . negateOuter . BI.unsafeDataAsMap
 where
  negateOuter  value =
    BI.caseList'
      BI.mkNilData                        
      (\pair restCS ->                                 
        let newPair =
              BI.casePair pair $ \(cs, tkMapData) ->
                BI.mkPairData cs (negateTokens $ BI.unsafeDataAsMap tkListData)
         in BI.mkCons newPair (negateOuter restCS)
      )
    value
  negateTokens tks =
      BI.caseList'
        BI.mkNilData                        
        (\tkPair restTk ->
            let newTkPair =
                  BI.casePair tkPair $ \(tkName, amtData) ->
                     BI.mkPairData tkName (BI.mkI $ 0 - (BI.unsafeDataAsI amtData))
             in BI.mkCons newTkPair (negateTokens restTk)
        )
        tks

The above blows up the budget when applied to even eight or nine values with less than ten unique tokens each. This alone makes it impossible for programmable tokens to go to production.

[zliu41]

caseList' and casePair currently expand to large and inefficient terms: https://github.com/IntersectMBO/plutus/blob/7ebd44f7c83ba4909a66ab3466802ad6b9714fd2/plutus-tx-plugin/src/PlutusTx/Compiler/Builtins.hs#L390-L527

After the HF, the terms under the _BuiltinCasing case will be used, which should be a lot cheaper. Of course, they may still not be cheap enough for your use case.